Repository class refactoring #95

arthurmco · 2025-01-27T14:11:03Z

This is a HUGE refactoring adding some structure to the code base (namespace and folders) and removing old RepoManager code

Generalized RepoManager over the repository type and the runner,
Removed RepoManger and Runner from cluster
Replace the old implementation of RepoManager with the new implementation
Remove the dependency Runner from Cluster, if something is depending on the cluster's runner to run commands we'll find out, this something should be running commands on its own
Added getRepoManager and getRunner for retrieving the respective singletons
I added namespaces cloyster::models and cloyster::services to start splitting the code according to the model-view-presenter architecture
I moved the files so they correspond to their namespaces
Refactored RepoManager to use glibmm KeyFile for parsing
Refactored RepoManager public API
Refactored RepoManager implementation
Removed XCAT code from repo manager
Added KeyFile implementation hiding glibmm KeyFile (files.h/files.cpp)
Added concepts.h header for general concepts
Standarise dry run logging messages
Isolate presenter and viewer in its own cmake target
Add CATTUS_SKIP_DISK_CHECKSUM environment to skip checksum of the ISO
Add --test-conf-file for testing KeyFile implementation against arbitrary files (it loads and prints the file)

src/services/repo.cpp

viniciusferrao · 2025-01-27T14:19:39Z

src/services/repo.cpp

+
+void ELRepoFile::read()
+{
+    printf("reading\n");


src/services/repo.cpp

include/cloysterhpc/services/repo.h

viniciusferrao · 2025-01-27T14:25:50Z

src/services/repo.cpp

+        m_repositories.push_back(std::move(repo));
+    }
+
+    for (const auto& r : m_repositories) {


I would avoid using only r as variable name. Usually we use it for well defined counters, like: i, j and k. Or for iterators, like it.

This is only code for toying with glibmm

In real life, this would not be like this

(This comment applies to every printf/fmt::format comment above)

viniciusferrao · 2025-01-27T14:27:43Z

Refactor the repository class based on @dhilst 's modeling

The modeling was done by the three of us. You included!

include/cloysterhpc/services/repo.h

dhilst · 2025-01-27T14:34:15Z

include/cloysterhpc/services/repo.h

+class ELRepoFile : public RepoFile<ELRepo> {
+private:
+    Glib::RefPtr<Glib::KeyFile> loadFile();
+
+public:
+    ELRepoFile(const std::filesystem::path& path)
+        : RepoFile(path)
+    {
+    }
+
+    void read() override;
+    void write() override;
+};


We can have an abstract RepoFile which extends the File, adds the m_repositories generic member, and with parse unparse methods, read is super read followed by parse and write is unparse followed by super write

std::filesystem::path is convertible to std::string or const char*. We must use it. It's the right API.

However the file class that be dependent on std::filesystem::path and not the derived one.

Just to be clear

parse loads data from the file into the m_repositories member

unparse loads the m_repositories member into the file

Also, @arthurmco, add overloads to parse and unparse which I/O into istringstream ostringstream instead of directly into the file so we can test without touching the disks. I don't know if this is possible because I don't know the interface of glibmm if it can load/unload keyfiles from strings, can it?

This is possible; glibmm can load keyhole data from a string.

include/cloysterhpc/services/repo.h

src/services/repo.cpp

dhilst · 2025-01-30T13:48:31Z

src/services/repo.cpp

+std::vector<ELRepo> ELRepoFile::parse(const std::stringstream& ss)
+{
+    m_file = Glib::KeyFile::create();
+    m_file->load_from_data(ss.str().c_str());
+    return this->parseData();
+}


Cool, now

Add m_repositories member as a vector of ELRepo with a getter

The getter should load the data into the member if the member is empty and return at the end

Inside of ELRepo?

Inside ELRepoFile

dhilst · 2025-01-30T13:50:52Z

include/cloysterhpc/services/repo.h

+struct ELRepo {
+    std::string group;
+    std::string name;
+    std::optional<std::string> baseURL;


include/cloysterhpc/services/repo.h

dhilst · 2025-01-30T14:04:20Z

src/services/repo.cpp

+    auto get_optional_string
+        = [this](auto group, auto key) -> std::optional<Glib::ustring> {
+        return m_file->has_key(group, key)
+            ? std::make_optional(m_file->get_string(group, key))
+            : std::nullopt;
+    };


I don't see why this captures this, anyway, move it to functions.cpp

dhilst

The API looks good, there are a few points

Make sure to remove the prints
Rename the ELReo to ELCloneRepo
Remove the PR from draft

viniciusferrao

Another issue that I've observed. The virtual classes should be on its own files. This needs to be fixed.

viniciusferrao · 2025-01-30T13:54:28Z

src/services/repo.cpp

+
+std::vector<ELRepo> ELRepoFile::parse()
+{
+    this->read();


Do we need "this"?

src/services/repo.cpp

viniciusferrao · 2025-01-31T14:55:44Z

include/cloysterhpc/functions.h

@@ -164,6 +166,9 @@ std::string findAndReplace(const std::string_view& source,
 */
 void copyFile(std::filesystem::path source, std::filesystem::path destination);

+std::optional<Glib::ustring> readKeyfileString(Glib::RefPtr<Glib::KeyFile> file,
+    const std::string_view& group, const std::string_view& key);


std:string_view is already a pointer to a string. It should not be passed as const ref.

I didn't know that. I thought it was pointer+length. It will be fixed

include/cloysterhpc/services/repo.h

viniciusferrao · 2025-02-01T02:03:58Z

src/functions.cpp

@@ -351,4 +351,12 @@ void copyFile(std::filesystem::path source, std::filesystem::path destination)
    }
 }

+std::optional<Glib::ustring> readKeyfileString(Glib::RefPtr<Glib::KeyFile> file,
+    const std::string_view& group, const std::string_view& key)


No need to be const ref.

See this comment

viniciusferrao

I ended up adding a lot of comments.

I think we should think a little bit more on this implementation.

viniciusferrao · 2025-02-07T01:09:08Z

include/cloysterhpc/file.h

The class name must match de filename. And also the #ifndef guards.

I would recommend using a namespace to avoid clashing.

viniciusferrao · 2025-02-07T01:15:30Z

include/cloysterhpc/file.h

+    virtual void read() { }
+    virtual void write() { }


This should be = 0? I can't find the implementation in other files, only the specialization. It should be pure virtual.

viniciusferrao · 2025-02-07T01:19:22Z

include/cloysterhpc/file.h

+    {
+    }
+
+    virtual void read() { }


I think read will not modify the state of the object right? Should it be const?

include/cloysterhpc/file.h

viniciusferrao · 2025-02-07T01:26:31Z

include/cloysterhpc/file.h

+    virtual void read() { }
+    virtual void write() { }
+
+    virtual ~GenericFile() = default;


Since you defaulted a destructor, AFAIK, the move operations are by default deleted. I think we should define the move operations:

GenericFile(GenericFile&&) = default; GenericFile& operator=(GenericFile&&) = default;

src/services/repofile.cpp

viniciusferrao · 2025-02-07T02:06:11Z

src/services/repofile.cpp

+        repo.enabled = enabled;
+        repo.gpgcheck = gpgcheck;
+        repo.gpgkey = gpgkey;
+        repositories.push_back(std::move(repo));


emplace_back.

src/services/repofile.cpp

viniciusferrao · 2025-02-07T02:11:25Z

src/services/repofile.cpp

+}
+
+[[nodiscard]] const std::vector<ELCloneRepo>&
+ELRepoFile::getRepositoriesConst() const


Having the qualifier in the method name is a very bad approach.

src/services/repofile.cpp

viniciusferrao

That's a big one. There's some comments that were probably not from this PR originally, we should mark those as a BUG to be fixed later.

For everything else can you evaluate @dhilst? Don't be economic when naming things, you can abuse it. Avoids guessing. The compiler will optimize anyway.

viniciusferrao · 2025-03-12T21:19:16Z

include/cloysterhpc/concepts.h

+#ifndef CLOYSTER_CONCEPTS_H
+#define CLOYSTER_CONCEPTS_H


Suggested change

#ifndef CLOYSTER_CONCEPTS_H

#define CLOYSTER_CONCEPTS_H

/*

* Copyright 2025 Vinícius Ferrão <vinicius@ferrao.net.br>

* SPDX-License-Identifier: Apache-2.0

*/

#ifndef CLOYSTERHPC_CONCEPTS_H_

#define CLOYSTERHPC_CONCEPTS_H_

viniciusferrao · 2025-03-12T21:19:57Z

include/cloysterhpc/concepts.h

+ * @brief IsParser<P, T> means: P can parse and unparse Ts from streams. The
+ * parsers are allowed to throw exceptions
+ */
+template <typename Parser_, typename I, typename O>


I is input and O is output? If yes I think we should use more descriptive names.

viniciusferrao · 2025-03-12T21:21:01Z

include/cloysterhpc/concepts.h

+ * The parsers are not allowed to throw exceptions, it must return errors of
+ * type E
+ */
+template <typename Parser_, typename T, typename E>


What is T and E?

Also the naming with _ at the end, is that a convention?

I removed this concept, anyway T is the parsed type and E the error type

viniciusferrao · 2025-03-12T21:21:20Z

include/cloysterhpc/concepts.h

+
+}
+
+#endif // CLOYSTER_CONCEPTS_H


Suggested change

#endif // CLOYSTER_CONCEPTS_H

#endif // CLOYSTERHPC_CONCEPTS_H_

viniciusferrao · 2025-03-12T22:09:39Z

include/cloysterhpc/diskImage.h

+        { "AlmaLinux-8.8-x86_64-dvd.iso", "OracleLinux-R8-U8-x86_64-dvd.iso",
+            "rhel-8.8-x86_64-dvd.iso", "Rocky-8.8-x86_64-dvd1.iso",
            "Rocky-9.5-x86_64-dvd.iso" }) };


I think this should be better organized or generated since it's a pattern and not hardcoded like that. I know it's a problem from legacy code. But maybe this should be removed.

src/services/repos.cpp

viniciusferrao · 2025-03-12T23:33:30Z

src/services/repos.cpp

+        case OS::Platform::el8:
+        case OS::Platform::el9:
+        case OS::Platform::el10:
+            return "/etc/yum.repos.d/cloyster.repo";


I think this will explicit return as const char*. Should we force it to return an std::string?

viniciusferrao · 2025-03-12T23:33:51Z

src/services/repos.cpp

+    std::vector<std::string> dependencies;
+
+    std::size_t version = osinfo.getMajorVersion();
+    std::string powertools = "powertools";


That logic is not good at all.

I didn't get it, how exactly it is not good?

viniciusferrao · 2025-03-12T23:34:35Z

src/services/repos.cpp

+void RepoManager::enable(const std::string& repoid)
+{
+    if (cloyster::dryRun) {
+        LOG_WARN("Dry Run: Would enable repository {}", repoid);


Should all dry run statements be LOG_INFO instead? It's not a warning.

This is because I standardized all the dry-run logging messages to be warning and have Dry Run: as the prefix, for the sake of consistency

viniciusferrao · 2025-03-12T23:35:43Z

src/services/shell.cpp

+            return std::vector<std::string>(
+                       { "-beegfs", "-elrepo", "-epel", "-openhpc",
+                           "-openhpc-updates", "-rpmfusion-free-updates" })


That's temporary?

It is not. These are the repositories enabled by default in EL distros, it comes from old/refactored code. Is it wrong?

* Add script to build rpms * Fix debug build * Link against host Glibmm using pkg-config * Extract glibmm from diskImage, move it to files.cpp --------- Co-authored-by: Daniel Hilst <daniel@versatushpc.com.br>

sonarqubecloud · 2025-03-17T17:15:48Z

Quality Gate failed

Failed conditions
20 Security Hotspots

See analysis details on SonarQube Cloud

dhilst · 2025-03-18T17:18:25Z

@viniciusferrao Changing the smart pointers in the postfix (for the client bus) requires refactoring and changing the IService interface. Replacing shared with unique pointers in the presenter requires a major refactoring too because these classes should not have ownership of these objects

I will address these problems in another PR

WIP: Repository class refactoring

a01df8c

arthurmco requested a review from dhilst January 27, 2025 14:11

arthurmco self-assigned this Jan 27, 2025

viniciusferrao requested changes Jan 27, 2025

View reviewed changes

dhilst reviewed Jan 27, 2025

View reviewed changes

include/cloysterhpc/services/repo.h Outdated Show resolved Hide resolved

dhilst reviewed Jan 27, 2025

View reviewed changes

include/cloysterhpc/services/repo.h Outdated Show resolved Hide resolved

dhilst reviewed Jan 27, 2025

View reviewed changes

src/services/repo.cpp Outdated Show resolved Hide resolved

arthurmco added 2 commits January 30, 2025 08:31

Add parse() and unparse() methods, fix more things

6fe6003

Add stringstream read specialization, as suggested by @dhilst

99f79ce

dhilst reviewed Jan 30, 2025

View reviewed changes

include/cloysterhpc/services/repo.h Outdated Show resolved Hide resolved

dhilst reviewed Jan 30, 2025

View reviewed changes

arthurmco added 3 commits January 31, 2025 09:38

Move some common functions to functions.h

3f0f219

Remove this->

37f8b60

Fix repo class

ab4ba11

dhilst approved these changes Jan 31, 2025

View reviewed changes

dhilst requested changes Jan 31, 2025

View reviewed changes

viniciusferrao requested changes Feb 1, 2025

View reviewed changes

arthurmco added 2 commits February 3, 2025 10:30

Fix const ref and declaration name

25df8ed

Fix naming

de966e5

arthurmco requested a review from dhilst February 4, 2025 13:44

arthurmco marked this pull request as ready for review February 4, 2025 13:45

arthurmco added 2 commits February 4, 2025 12:37

Rename required files

4895e2e

fixup! Rename required files

a834f94

viniciusferrao requested changes Feb 7, 2025

View reviewed changes

dhilst assigned dhilst and unassigned arthurmco Feb 7, 2025

Refactoring RepoManager

1c1ac70

dhilst force-pushed the repo-class branch 4 times, most recently from c00412d to 79954f1 Compare February 18, 2025 03:46

Testing RepoManager refactoring

924dcb1

dhilst force-pushed the repo-class branch from 79954f1 to 924dcb1 Compare February 18, 2025 14:19

Daniel Hilst added 2 commits February 18, 2025 11:51

Run clang-format

e0121c6

Testing release build

c4623cb

dhilst force-pushed the repo-class branch from 169037c to c4623cb Compare February 19, 2025 19:23

dhilst marked this pull request as ready for review February 19, 2025 20:21

Daniel Hilst added 3 commits February 19, 2025 17:21

Run clang-format again

59d485e

Remove empty file

0023eaf

Remove commented code

9cefe91

dhilst changed the title ~~WIP: Repository class refactoring~~ Repository class refactoring Feb 19, 2025

dhilst force-pushed the repo-class branch from 398f732 to 49c5382 Compare February 20, 2025 03:31

Add RPMRepoManager sketch implementation

9d58572

dhilst force-pushed the repo-class branch from 49c5382 to 9d58572 Compare February 20, 2025 13:40

Daniel Hilst added 2 commits February 24, 2025 18:07

Replace old code with RPMRepositoryManager in repos.cpp

9072ce4

Remove unused code

17de642

dhilst force-pushed the repo-class branch 2 times, most recently from 9d0b02b to ecb7a02 Compare February 25, 2025 00:45

Save RPMRepositoryFile only once for multiple enable calls

a696419

dhilst force-pushed the repo-class branch from ecb7a02 to a696419 Compare February 25, 2025 01:08

Run clang-format

9e9a860

viniciusferrao requested changes Mar 12, 2025

View reviewed changes

RPM Build script && Use host glibmm (#96)

ae1436e

* Add script to build rpms * Fix debug build * Link against host Glibmm using pkg-config * Extract glibmm from diskImage, move it to files.cpp --------- Co-authored-by: Daniel Hilst <daniel@versatushpc.com.br>

dhilst mentioned this pull request Mar 24, 2025

RPM Build script && Use host glibmm #96

Merged

-#ifndef CLOYSTER_CONCEPTS_H
-#define CLOYSTER_CONCEPTS_H
+/*
+ * Copyright 2025 Vinícius Ferrão <vinicius@ferrao.net.br>
+ * SPDX-License-Identifier: Apache-2.0
+ */
+#ifndef CLOYSTERHPC_CONCEPTS_H_
+#define CLOYSTERHPC_CONCEPTS_H_

	#endif // CLOYSTER_CONCEPTS_H
	#endif // CLOYSTERHPC_CONCEPTS_H_

Repository class refactoring #95

Are you sure you want to change the base?

Repository class refactoring #95

Conversation

arthurmco commented Jan 27, 2025 • edited by dhilst Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

viniciusferrao commented Jan 27, 2025

dhilst Jan 27, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dhilst Jan 27, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dhilst Jan 30, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dhilst left a comment

Choose a reason for hiding this comment

viniciusferrao left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

viniciusferrao left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

viniciusferrao left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sonarqubecloud bot commented Mar 17, 2025

Quality Gate failed

dhilst commented Mar 18, 2025

arthurmco commented Jan 27, 2025 •

edited by dhilst

Loading

dhilst Jan 27, 2025 •

edited

Loading

dhilst Jan 27, 2025 •

edited

Loading

dhilst Jan 30, 2025 •

edited

Loading