OpenCTI #36

dial647s · 2025-02-06T04:37:19Z

I have a local OpenCTI deployment and which is authenticated using username/password. Is it possible to user password based auth instead of API?

dial647s · 2025-02-06T10:30:50Z

my bad.. figured out the API key and entered the details in the app. Ran a search for an observable available in OpenCTI but its been analysing forever. Not sure what the problem is.

stanfrbd · 2025-02-06T10:41:49Z

Hello @dial647s

which value did you put for opencti_url in secrets.json ?

Is there any error in the logs?

dial647s · 2025-02-06T10:52:33Z

oops. figured out the issue. The app was trying to reach OpenCTI over the internet. Changed that to private IP and now the response is snappy. However, I only see the searched telemetry in the results page. Will it pull context and display as well?

stanfrbd · 2025-02-06T10:58:17Z

@dial647s the OpenCTI connector tries to retrieve data and if it finds something, it gives the link, the score of the latest IoC and so on. But if nothing is found, you won't have much.

For example (on Cyberbro demo): when I'm looking for 1.1.1.1 I won't find related indicators, however I will get the stats of entities:

But if I'm looking for a real (and existing) indicator, I will have the following:

dial647s · 2025-02-06T11:02:34Z

Thank you mate. Its working fine now. Issue was with the DNS entry. BTW, I quite often get "Internal server error" when running a search however it resolves upon hitting refresh once or twice. Anyway to overcome this error?

stanfrbd · 2025-02-06T11:06:17Z

Do you have a reverse proxy?
Maybe it's about hardware resources, can you give me your hardware specs for your installation?
Maybe try to modify your gunicorn threads and workers.

You can follow the steps here https://github.com/stanfrbd/cyberbro/wiki/4.-Advanced-options-in-secrets.json to change the number of threads (1) and workers (1) and rebuild the image after (docker compose down && docker compose up -d --build)

dial647s · 2025-02-06T11:12:03Z

No reverse proxy. I am running this on my AWS EC2 instance.. running Amazon Linux 2023, t3.xlarge with 16gb memory. Let me try the changes recommended in that doc and get back.

dial647s · 2025-02-06T11:21:54Z

well, still getting internet server error after making the change and re-initialising the container as you suggested. Lets see.

stanfrbd · 2025-02-06T11:26:53Z

Hum what do the logs say?

Connect to the container

docker exec -it cyberbro bash

and

tail -f /var/log/cyberbro/*.log

dial647s · 2025-02-06T11:37:05Z

==> /var/log/cyberbro/cyberbro.err.log <==
d = self.chunk()
File "/usr/local/lib/python3.13/site-packages/gunicorn/http/unreader.py", line 63, in chunk
return self.sock.recv(self.mxchunk)
~~~~~~~~~~~~~~^^^^^^^^^^^^^^
File "/usr/local/lib/python3.13/site-packages/gunicorn/workers/base.py", line 204, in handle_abort
sys.exit(1)
~~~~~~~~^^^
SystemExit: 1
[2025-02-06 11:20:27 +0000] [11] [INFO] Worker exiting (pid: 11)
[2025-02-06 11:20:28 +0000] [56] [INFO] Booting worker with pid: 56

==> /var/log/cyberbro/cyberbro.out.log <==

==> /var/log/cyberbro/cyberbro.err.log <==
[2025-02-06 11:34:42 +0000] [7] [CRITICAL] WORKER TIMEOUT (pid:34)
[2025-02-06 11:34:42 +0000] [34] [ERROR] Error handling request (no URI read)
Traceback (most recent call last):
File "/usr/local/lib/python3.13/site-packages/gunicorn/workers/sync.py", line 133, in handle
req = next(parser)
File "/usr/local/lib/python3.13/site-packages/gunicorn/http/parser.py", line 41, in next
self.mesg = self.mesg_class(self.cfg, self.unreader, self.source_addr, self.req_count)
~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.13/site-packages/gunicorn/http/message.py", line 259, in init
super().init(cfg, unreader, peer_addr)
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.13/site-packages/gunicorn/http/message.py", line 60, in init
unused = self.parse(self.unreader)
File "/usr/local/lib/python3.13/site-packages/gunicorn/http/message.py", line 271, in parse
self.get_data(unreader, buf, stop=True)
~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.13/site-packages/gunicorn/http/message.py", line 262, in get_data
data = unreader.read()
File "/usr/local/lib/python3.13/site-packages/gunicorn/http/unreader.py", line 36, in read
d = self.chunk()
File "/usr/local/lib/python3.13/site-packages/gunicorn/http/unreader.py", line 63, in chunk
return self.sock.recv(self.mxchunk)
~~~~~~~~~~~~~~^^^^^^^^^^^^^^
File "/usr/local/lib/python3.13/site-packages/gunicorn/workers/base.py", line 204, in handle_abort
sys.exit(1)
~~~~~~~~^^^
SystemExit: 1
[2025-02-06 11:34:42 +0000] [34] [INFO] Worker exiting (pid: 34)
[2025-02-06 11:34:42 +0000] [71] [INFO] Booting worker with pid: 71
Could not extract a valid registered domain from '193.233.132.177'.
Error resolving reverse DNS for '193.233.132.177' (IPv4): The DNS query name does not exist: 177.132.233.193.in-addr.arpa.
Traceback (most recent call last):
File "/app/engines/reverse_dns.py", line 34, in reverse_dns
answer = dns.resolver.resolve(reverse_name, "PTR")
File "/usr/local/lib/python3.13/site-packages/dns/resolver.py", line 1564, in resolve
return get_default_resolver().resolve(
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
qname,
^^^^^^
...<7 lines>...
search,
^^^^^^^
)
^
File "/usr/local/lib/python3.13/site-packages/dns/resolver.py", line 1306, in resolve
(request, answer) = resolution.next_request()
~~~~~~~~~~~~~~~~~~~~~~~^^
File "/usr/local/lib/python3.13/site-packages/dns/resolver.py", line 750, in next_request
raise NXDOMAIN(qnames=self.qnames_to_try, responses=self.nxdomain_responses)
dns.resolver.NXDOMAIN: The DNS query name does not exist: 177.132.233.193.in-addr.arpa.
Error resolving reverse DNS for 'http://193.233.132.177/lbbb.exe' (URL): The DNS query name does not exist: 177.132.233.193.in-addr.arpa.
Traceback (most recent call last):
File "/app/engines/reverse_dns.py", line 64, in reverse_dns
answer = dns.resolver.resolve(reverse_name, "PTR")
File "/usr/local/lib/python3.13/site-packages/dns/resolver.py", line 1564, in resolve
return get_default_resolver().resolve(
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
qname,
^^^^^^
...<7 lines>...
search,
^^^^^^^
)
^
File "/usr/local/lib/python3.13/site-packages/dns/resolver.py", line 1306, in resolve
(request, answer) = resolution.next_request()
~~~~~~~~~~~~~~~~~~~~~~~^^
File "/usr/local/lib/python3.13/site-packages/dns/resolver.py", line 750, in next_request
raise NXDOMAIN(qnames=self.qnames_to_try, responses=self.nxdomain_responses)
dns.resolver.NXDOMAIN: The DNS query name does not exist: 177.132.233.193.in-addr.arpa.
[2025-02-06 11:35:32 +0000] [7] [CRITICAL] WORKER TIMEOUT (pid:56)
[2025-02-06 11:35:32 +0000] [56] [ERROR] Error handling request (no URI read)
Traceback (most recent call last):
File "/usr/local/lib/python3.13/site-packages/gunicorn/workers/sync.py", line 133, in handle
req = next(parser)
File "/usr/local/lib/python3.13/site-packages/gunicorn/http/parser.py", line 41, in next
self.mesg = self.mesg_class(self.cfg, self.unreader, self.source_addr, self.req_count)
~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.13/site-packages/gunicorn/http/message.py", line 259, in init
super().init(cfg, unreader, peer_addr)
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.13/site-packages/gunicorn/http/message.py", line 60, in init
unused = self.parse(self.unreader)
File "/usr/local/lib/python3.13/site-packages/gunicorn/http/message.py", line 271, in parse
self.get_data(unreader, buf, stop=True)
~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.13/site-packages/gunicorn/http/message.py", line 262, in get_data
data = unreader.read()
File "/usr/local/lib/python3.13/site-packages/gunicorn/http/unreader.py", line 36, in read
d = self.chunk()
File "/usr/local/lib/python3.13/site-packages/gunicorn/http/unreader.py", line 63, in chunk
return self.sock.recv(self.mxchunk)
~~~~~~~~~~~~~~^^^^^^^^^^^^^^
File "/usr/local/lib/python3.13/site-packages/gunicorn/workers/base.py", line 204, in handle_abort
sys.exit(1)
~~~~~~~~^^^
SystemExit: 1
[2025-02-06 11:35:32 +0000] [56] [INFO] Worker exiting (pid: 56)
[2025-02-06 11:35:33 +0000] [78] [INFO] Booting worker with pid: 78

dial647s · 2025-02-06T11:40:24Z

==> /var/log/cyberbro/cyberbro.err.log <==
d = self.chunk()
File "/usr/local/lib/python3.13/site-packages/gunicorn/http/unreader.py", line 63, in chunk
return self.sock.recv(self.mxchunk)
~~~~~~~~~~~~~~^^^^^^^^^^^^^^
File "/usr/local/lib/python3.13/site-packages/gunicorn/workers/base.py", line 204, in handle_abort
sys.exit(1)
~~~~~~~~^^^
SystemExit: 1
[2025-02-06 11:35:32 +0000] [56] [INFO] Worker exiting (pid: 56)
[2025-02-06 11:35:33 +0000] [78] [INFO] Booting worker with pid: 78

==> /var/log/cyberbro/cyberbro.out.log <==

==> /var/log/cyberbro/cyberbro.err.log <==
[2025-02-06 11:39:07 +0000] [7] [CRITICAL] WORKER TIMEOUT (pid:78)
[2025-02-06 11:39:07 +0000] [78] [ERROR] Error handling request (no URI read)
Traceback (most recent call last):
File "/usr/local/lib/python3.13/site-packages/gunicorn/workers/sync.py", line 133, in handle
req = next(parser)
File "/usr/local/lib/python3.13/site-packages/gunicorn/http/parser.py", line 41, in next
self.mesg = self.mesg_class(self.cfg, self.unreader, self.source_addr, self.req_count)
~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.13/site-packages/gunicorn/http/message.py", line 259, in init
super().init(cfg, unreader, peer_addr)
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.13/site-packages/gunicorn/http/message.py", line 60, in init
unused = self.parse(self.unreader)
File "/usr/local/lib/python3.13/site-packages/gunicorn/http/message.py", line 271, in parse
self.get_data(unreader, buf, stop=True)
~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.13/site-packages/gunicorn/http/message.py", line 262, in get_data
data = unreader.read()
File "/usr/local/lib/python3.13/site-packages/gunicorn/http/unreader.py", line 36, in read
d = self.chunk()
File "/usr/local/lib/python3.13/site-packages/gunicorn/http/unreader.py", line 63, in chunk
return self.sock.recv(self.mxchunk)
~~~~~~~~~~~~~~^^^^^^^^^^^^^^
File "/usr/local/lib/python3.13/site-packages/gunicorn/workers/base.py", line 204, in handle_abort
sys.exit(1)
~~~~~~~~^^^
SystemExit: 1
[2025-02-06 11:39:07 +0000] [78] [INFO] Worker exiting (pid: 78)
[2025-02-06 11:39:07 +0000] [83] [INFO] Booting worker with pid: 83
[2025-02-06 11:39:34 +0000] [7] [CRITICAL] WORKER TIMEOUT (pid:71)
[2025-02-06 11:39:34 +0000] [71] [ERROR] Error handling request (no URI read)
Traceback (most recent call last):
File "/usr/local/lib/python3.13/site-packages/gunicorn/workers/sync.py", line 133, in handle
req = next(parser)
File "/usr/local/lib/python3.13/site-packages/gunicorn/http/parser.py", line 41, in next
self.mesg = self.mesg_class(self.cfg, self.unreader, self.source_addr, self.req_count)
~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.13/site-packages/gunicorn/http/message.py", line 259, in init
super().init(cfg, unreader, peer_addr)
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.13/site-packages/gunicorn/http/message.py", line 60, in init
unused = self.parse(self.unreader)
File "/usr/local/lib/python3.13/site-packages/gunicorn/http/message.py", line 271, in parse
self.get_data(unreader, buf, stop=True)
~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.13/site-packages/gunicorn/http/message.py", line 262, in get_data
data = unreader.read()
File "/usr/local/lib/python3.13/site-packages/gunicorn/http/unreader.py", line 36, in read
d = self.chunk()
File "/usr/local/lib/python3.13/site-packages/gunicorn/http/unreader.py", line 63, in chunk
return self.sock.recv(self.mxchunk)
~~~~~~~~~~~~~~^^^^^^^^^^^^^^
File "/usr/local/lib/python3.13/site-packages/gunicorn/workers/base.py", line 204, in handle_abort
sys.exit(1)
~~~~~~~~^^^
SystemExit: 1
[2025-02-06 11:39:34 +0000] [71] [INFO] Worker exiting (pid: 71)
[2025-02-06 11:39:35 +0000] [89] [INFO] Booting worker with pid: 89

stanfrbd · 2025-02-06T12:56:08Z

It seems there is an issue with gunicorn, is secrets.json present in the directory?

I suggest you to stop the container, then start it again (with a previous docker system prune - careful, it will delete all unused images / containers / networks) to see if fixes the issue.

dial647s · 2025-02-06T20:42:16Z

Yes, secrets.json in the same folder.. it works with a refresh.
Did a system prune and restarted the instance as suggested.. same results. logs as follows.

[2025-02-06 20:39:56 +0000] [46] [INFO] Booting worker with pid: 46
[2025-02-06 20:40:07 +0000] [7] [CRITICAL] WORKER TIMEOUT (pid:46)
[2025-02-06 20:40:07 +0000] [46] [ERROR] Error handling request (no URI read)
Traceback (most recent call last):
File "/usr/local/lib/python3.13/site-packages/gunicorn/workers/sync.py", line 133, in handle
req = next(parser)
File "/usr/local/lib/python3.13/site-packages/gunicorn/http/parser.py", line 41, in next
self.mesg = self.mesg_class(self.cfg, self.unreader, self.source_addr, self.req_count)
~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.13/site-packages/gunicorn/http/message.py", line 259, in init
super().init(cfg, unreader, peer_addr)
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.13/site-packages/gunicorn/http/message.py", line 60, in init
unused = self.parse(self.unreader)
File "/usr/local/lib/python3.13/site-packages/gunicorn/http/message.py", line 271, in parse
self.get_data(unreader, buf, stop=True)
~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.13/site-packages/gunicorn/http/message.py", line 262, in get_data
data = unreader.read()
File "/usr/local/lib/python3.13/site-packages/gunicorn/http/unreader.py", line 36, in read
d = self.chunk()
File "/usr/local/lib/python3.13/site-packages/gunicorn/http/unreader.py", line 63, in chunk
return self.sock.recv(self.mxchunk)
~~~~~~~~~~~~~~^^^^^^^^^^^^^^
File "/usr/local/lib/python3.13/site-packages/gunicorn/workers/base.py", line 204, in handle_abort
sys.exit(1)
~~~~~~~~^^^
SystemExit: 1
[2025-02-06 20:40:07 +0000] [46] [INFO] Worker exiting (pid: 46)
[2025-02-06 20:40:08 +0000] [50] [INFO] Booting worker with pid: 50

stanfrbd · 2025-02-06T20:48:40Z

I think it may be about the IP direct access without reverse proxy, I think that gunicorn doesn't like it very much (it's not a problem when it is with localhost, but It crashes too when I try with other internal IP e.g. 192.168.1.X or public IP direct access).

Can you try to add a reverse proxy like caddy or traefik?

You can just do the command reverse proxy to http://your_IP:5000

dial647s · 2025-02-06T21:11:15Z

Thanks, I'll give it a go and see if it works. Will keep you posted.

stanfrbd · 2025-02-06T21:18:20Z

It seems the problem happens often with the timeout

https://stackoverflow.com/questions/10855197/gunicorn-how-to-resolve-worker-timeout

This link references it, we must add a timeout of 120 for cloud stuff I guess. I suggest you edit the file supervisord.conf and add --timeout 120 to the gunicorn command.

Then you can redeploy your container.

Let me know if it helps!

stanfrbd · 2025-02-07T10:00:34Z

@dial647s I released v0.4.0 with default 120 timeout, update and tell me if it changes something.

dial647s · 2025-02-07T10:36:10Z

@stanfrbd - deployed the new version.. works like a charm.. thank you.
PS: Do i remove the following from the secrets.json?

"supervisord_workers_count": 1,
"supervisord_threads_count": 1

stanfrbd · 2025-02-07T10:37:26Z

@dial647s you can keep these settings if you are a single user :)

Actually the 120 did not apply properly so I changed the timeout explicitely, you can update again if you have still issues.

stanfrbd added the question Further information is requested label Feb 7, 2025

stanfrbd self-assigned this Feb 7, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

OpenCTI #36

OpenCTI #36

dial647s commented Feb 6, 2025

dial647s commented Feb 6, 2025

stanfrbd commented Feb 6, 2025 •

edited

Loading

dial647s commented Feb 6, 2025

stanfrbd commented Feb 6, 2025

dial647s commented Feb 6, 2025

stanfrbd commented Feb 6, 2025 •

edited

Loading

dial647s commented Feb 6, 2025

dial647s commented Feb 6, 2025

stanfrbd commented Feb 6, 2025

dial647s commented Feb 6, 2025

dial647s commented Feb 6, 2025

stanfrbd commented Feb 6, 2025

dial647s commented Feb 6, 2025

stanfrbd commented Feb 6, 2025 •

edited

Loading

dial647s commented Feb 6, 2025

stanfrbd commented Feb 6, 2025 •

edited

Loading

stanfrbd commented Feb 7, 2025 •

edited

Loading

dial647s commented Feb 7, 2025

stanfrbd commented Feb 7, 2025 •

edited

Loading

OpenCTI #36

OpenCTI #36

Comments

dial647s commented Feb 6, 2025

dial647s commented Feb 6, 2025

stanfrbd commented Feb 6, 2025 • edited Loading

dial647s commented Feb 6, 2025

stanfrbd commented Feb 6, 2025

dial647s commented Feb 6, 2025

stanfrbd commented Feb 6, 2025 • edited Loading

dial647s commented Feb 6, 2025

dial647s commented Feb 6, 2025

stanfrbd commented Feb 6, 2025

dial647s commented Feb 6, 2025

dial647s commented Feb 6, 2025

stanfrbd commented Feb 6, 2025

dial647s commented Feb 6, 2025

stanfrbd commented Feb 6, 2025 • edited Loading

dial647s commented Feb 6, 2025

stanfrbd commented Feb 6, 2025 • edited Loading

stanfrbd commented Feb 7, 2025 • edited Loading

dial647s commented Feb 7, 2025

stanfrbd commented Feb 7, 2025 • edited Loading

stanfrbd commented Feb 6, 2025 •

edited

Loading

stanfrbd commented Feb 6, 2025 •

edited

Loading

stanfrbd commented Feb 6, 2025 •

edited

Loading

stanfrbd commented Feb 6, 2025 •

edited

Loading

stanfrbd commented Feb 7, 2025 •

edited

Loading

stanfrbd commented Feb 7, 2025 •

edited

Loading