List of CT Logs used

[uggyuggy]

Hi Rico,

I see the list of CT logs server used, when the daemon start

Starting worker for CT log: https://xxxx.sectigo.com
...

I have indeed the same number of CT log servers compared to those in Chrome: https://www.gstatic.com/ct/log_list/v3/log_list.json

To not miss any certificate, daemon needs to "track" all available CT log servers.
I'm wondering if that "list" of available CT log servers is updated only when the daemon starts ?
(From my logs, it looks like so, but wanted to confirm).

My concern is that if the daemon run smoothly during a long period of time without restart, could we "miss" some new CT log servers available, and miss some certificates ?

Thank's

[d-Rickyy-b]

Hi, your concern is totally valid. I thought about implementing this in the past but I seem to have forgotten about it. I'll add a background job that updates the logs from the ct log list every now and then. See also #42. What do you think is a good interval? I thought about checking once every 6 hours. That would be 4 times per day.

Also not sure about it, but the original server from calidog doesn't seem to support auto updates to the ct log list either, does it? My elixir knowledge is pretty limited, but it seems that it also only fetches the full list once at startup.

https://github.com/CaliDog/certstream-server/blob/60de7000901e5eb246d2e83c908678b43e5a60c8/lib/certstream/ct_watcher.ex#L22-L29

[d-Rickyy-b]

I pushed an update to master. See also v1.7.0-rc or diff. I'd appreciate some feedback and testing.

[uggyuggy]

Hi @d-Rickyy-b

First, thank you very much for the very fast feedback and new release 👌

What do you think is a good interval? I thought about checking once every 6 hours. That would be 4 times per day.

If I have to choose myself, I think I would pick every "1 hour" interval because:

• It seems like a very "light" operation that should not "load" the local nor the remote server
• 1 hour sounds less "strange" than a 6 hours value.
• Would be faster to check logs if the new feature is actually working ;)
• We would have less chance to miss any certificate compared to 6 hours
• Less than 1 hour would be overkill

An alternative would be to have this setting added to the config.yaml but I think that's not needed.

Also not sure about it, but the original server from calidog doesn't seem to support auto updates to the ct log list either, does it?

I don't remember is does this indeed..

I pushed an update to master. [...] I'd appreciate some feedback and testing.

Great ! Thank you !
I just updated to the v1.7.0-rc

• Server still works 😅
• I see some good things in logs

Found 47 new ct logs
Checking for new ct logs...
Found 0 new ct logs

I will check logs again after 6 hours of running 😉

[d-Rickyy-b]

• It seems like a very "light" operation that should not "load" the local nor the remote server
• 1 hour sounds less "strange" than a 6 hours value.
• Would be faster to check logs if the new feature is actually working ;)
• We would have less chance to miss any certificate compared to 6 hours
• Less than 1 hour would be overkill

Valid points. I updated it to run once per hour: 9b6e77d.

An alternative would be to have this setting added to the config.yaml but I think that's not needed.

It surely isn't difficult to add this to the config, but I'd rather want to keep the config file clean and not give too many choices that might be not too relevant. If there's demand for it, we can still add it at a later point.

Let me know if everything works. If there's no negative feedback, I'll push the 1.7.0 version at the start of next week.

[uggyuggy]

Hi Rico,

I confirm I see in logs with v1.7.0-rc, every 6 hours the log:

Checking for new ct logs...
Found 0 new ct logs

So looks good to me 👍 ( as much as we can test without a new CT log server arriving for real )

I updated it to run once per hour

Thanks 👍

If there's no negative feedback, I'll push the 1.7.0 version at the start of next week.

Sounds good, thank you 👍
I will test it when released.

[uggyuggy]

I may suggest 2 minor things for logs

• Adding a log line every hours after it tested for new ct logs... to summarize how many CT log servers are currenlty "known" or "in use"... CT logs currently in use: 47

Example:

Found 47 new ct logs   # Daemon start
...
...
Checking for new ct logs...
Found 0 new ct logs
CT logs currently in use: 47
...
Checking for new ct logs...
Found 0 new ct logs
CT logs currently in use: 47

• Having the number of CT logs at the end of the line
  so It could be easier to grep those lines in logs

New ct logs found: 47 instead of Found 47 new ct logs

Example

New CT logs servers found: 47  # Daemon start
...
Checking for new CT logs servers...
New CT logs servers found: 0
CT logs servers currently in use: 47
...
Checking for new CT logs servers...
New CT logs servers found: 0
CT logs servers currently in use: 47

$ grep "^CT logs servers currently in use:" foo.log

[d-Rickyy-b]

I implemented your suggestions and issued a new release: https://github.com/d-Rickyy-b/certstream-server-go/releases/tag/v1.7.0

Feel free to create a new issue/discussion if there's still open questions or issues.

[d-Rickyy-b]

Issue/suggestion was fixed and is included in this release: https://github.com/d-Rickyy-b/certstream-server-go/releases/tag/v1.7.0

[uggyuggy]

Thank you very much... New version installed 👍