Skip to content
This repository has been archived by the owner on Feb 8, 2024. It is now read-only.

CORTX-30044 Redirect hare log messages to stdout/stderr #2148

Merged
merged 1 commit into from
Aug 22, 2022
Merged

CORTX-30044 Redirect hare log messages to stdout/stderr #2148

merged 1 commit into from
Aug 22, 2022

Conversation

seagate-sarang-sawant
Copy link
Contributor

Problem: Redirect Error and Warning messages of Hare processes to stdout/stderr

Solution: Add stream handler to processes missing it. Stream handlers would
have INFO level logging with process id or thread added in log message
depending on context. Log level of some messages are changed to info
from debug.

Signed-off-by: Sarang Sawant sarang.sawant@seagate.com

@cla-bot cla-bot bot added the cla-signed label Aug 12, 2022
@pavankrishnat
Copy link
Contributor

retest this please

@mssawant
Copy link

mssawant commented Aug 16, 2022 •
edited
Loading

@seagate-sarang-sawant, please fix the commit message with respect to following points,

CORTX-30044 Redirect hare log messages to stdout/stderr

Problem: Redirect Error and Warning messages of Hare processes to stdout/stderr

Solution: Add stream handler to processes missing it. Stream handlers would
have INFO level logging with process id or thread added in log message
depending on context. Log level of some messages are changed to info
from debug.
  1. Problem needs to be a problem not a solution, in this case the problem statement appears to be a solution statement.
  2. Avoid capitalization post : (colon) this is debatable, but depends if you are introducing a statement or a list, but typically a letter post colon is not capitalized, https://www.sussex.ac.uk/informatics/punctuation/colonandsemi/colon.
  3. Itemize the problem and solution descriptions in case of multiple reasons.
  4. Need not duplicate the summary in problem statement if there's nothing additional to describe with respect to the problem.

mssawant
mssawant reviewed Aug 16, 2022
View reviewed changes
@seagate-sarang-sawant
CORTX-30044 Redirect hare log messages to stdout/stderr
b71c2cf 
Problem: a) kubectl logs display debug message and stream handlers
are missing from some hare processes.
b) log level is incorrect for some log messages in cfgen and hax

Solution: a) stream handlers added to cfgen, cli, update_conf,
consul and hax starters with info as default log level.
b) Stream handlers would have INFO level logging with format
change like process id or thread added in log message depending
on context.Log level of some messages in cfgen are changed
to info from debug.

Signed-off-by: Sarang Sawant <sarang.sawant@seagate.com>
@mssawant
Copy link

@seagate-sarang-sawant, so now do you see the info messages in kubectl logs of hax container? Can you post the output please.

mssawant
mssawant approved these changes Aug 20, 2022
View reviewed changes
Copy link

@mssawant mssawant left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good in general. Please post the output of kubectl logs as requested. Thank you.

@seagate-sarang-sawant
Copy link
Contributor Author

Kubectl logs for cortx-hax, cortx-setup containers

[root@ssc-vm-rhev4-2619 ~]# kubectl logs -f -c cortx-hax cortx-data-g0-0
2022-08-18 12:42:25,231 [INFO] Starting Hare services
2022-08-18 12:42:25,298 [INFO] Entering logrotate_generic at line 161 in file /opt/seagate/cortx/hare/lib/python3.6/site-packages/hare_mp/main.py
2022-08-18 12:42:25,299 [INFO] Entering get_log_dir at line 697 in file /opt/seagate/cortx/hare/lib/python3.6/site-packages/hare_mp/main.py
2022-08-18 12:42:25,378 [INFO] Leaving get_log_dir
2022-08-18 12:42:25,379 [INFO] Leaving logrotate_generic
2022-08-18 12:42:25,385 [INFO] Entering start_hax_and_consul_without_systemd at line 415 in file /opt/seagate/cortx/hare/lib/python3.6/site-packages/hare_mp/main.py
2022-08-18 12:42:25,386 [INFO] Entering get_config_dir at line 705 in file /opt/seagate/cortx/hare/lib/python3.6/site-packages/hare_mp/main.py
2022-08-18 12:42:25,449 [INFO] Leaving get_config_dir
2022-08-18 12:42:25,449 [INFO] Entering get_log_dir at line 697 in file /opt/seagate/cortx/hare/lib/python3.6/site-packages/hare_mp/main.py
2022-08-18 12:42:25,509 [INFO] Leaving get_log_dir
2022-08-18 12:42:25,509 [INFO] Entering _start_consul at line 240 in file /opt/seagate/cortx/hare/lib/python3.6/site-packages/hare_mp/main.py
2022-08-18 12:42:25,572 [INFO] Entering Utils.get_local_hostname at line 91 in file /opt/seagate/cortx/hare/lib/python3.6/site-packages/hare_mp/utils.py
2022-08-18 12:42:25,572 [INFO] Entering Utils.get_hostname at line 79 in file /opt/seagate/cortx/hare/lib/python3.6/site-packages/hare_mp/utils.py
2022-08-18 12:42:25,572 [INFO] Leaving Utils.get_hostname
2022-08-18 12:42:25,572 [INFO] Leaving Utils.get_local_hostname
2022-08-18 12:42:25,573 [INFO] Entering ConsulStarter._execute at line 66 in file /opt/seagate/cortx/hare/lib/python3.6/site-packages/hare_mp/consul_starter.py
2022-08-18 12:42:30,613 [INFO] Leaving _start_consul
2022-08-18 12:42:30,613 [INFO] Entering _start_hax at line 299 in file /opt/seagate/cortx/hare/lib/python3.6/site-packages/hare_mp/main.py
2022-08-18 12:42:30,614 [INFO] Entering HaxStarter._execute at line 54 in file /opt/seagate/cortx/hare/lib/python3.6/site-packages/hare_mp/hax_starter.py
2022-08-18 12:42:30,614 [INFO] Leaving _start_hax

[root@ssc-vm-rhev4-2619 ~]# kubectl logs -c cortx-setup cortx-data-g0-0
log: cortx_setup config apply -f yaml:///etc/cortx/solution/cluster.yaml -c 'yaml:///etc/cortx/cluster.conf'
2022-08-18 12:40:59 cortx_setup [19]: INFO [config_apply] Applying config yaml:///etc/cortx/solution/cluster.yaml
log: cortx_setup config apply -f yaml:///etc/cortx/solution/config.yaml -c 'yaml:///etc/cortx/cluster.conf'
2022-08-18 12:41:01 cortx_setup [42]: INFO [config_apply] Applying config yaml:///etc/cortx/solution/config.yaml
2022-08-18 12:41:01 cortx_setup [42]: WARNING [save] Release key cortx>common>release>name is missing or has incorrect value.
2022-08-18 12:41:01 cortx_setup [42]: INFO [save] Adding key "cortx>common>release>name" and value "CORTX" in confstore.
2022-08-18 12:41:01 cortx_setup [42]: WARNING [save] Release key cortx>common>release>version is missing or has incorrect value.
2022-08-18 12:41:01 cortx_setup [42]: INFO [save] Adding key "cortx>common>release>version" and value "2.0.0-7581" in confstore.
log: cortx_setup cluster bootstrap -c 'yaml:///etc/cortx/cluster.conf'
2022-08-18 12:41:15 cortx_setup [43]: INFO [_validate_provisioning_status] Recent phase for this node is None and recent status is None. Performing deployment on this node.
2022-08-18 12:41:15 cortx_setup [43]: INFO [cluster_deploy] Starting cluster bootstrap on 7ba8990a85d38508ca9bac4ff516d3ea:cortx-data-g0-0
2022-08-18 12:41:16 cortx_setup [43]: INFO [_provision_components] /opt/seagate/cortx/utils/bin/utils_setup post_install --config yaml:///etc/cortx/cluster.conf --services all
2022-08-18 12:41:17 cortx_setup [43]: INFO [_provision_components] /opt/seagate/cortx/motr/bin/motr_setup post_install --config yaml:///etc/cortx/cluster.conf --services io
2022-08-18 12:41:18 cortx_setup [43]: INFO [_provision_components] /opt/seagate/cortx/hare/bin/hare_setup post_install --config yaml:///etc/cortx/cluster.conf --services all
2022-08-18 12:41:20 cortx_setup [43]: INFO [_provision_components] /opt/seagate/cortx/utils/bin/utils_setup prepare --config yaml:///etc/cortx/cluster.conf --services all
2022-08-18 12:41:21 cortx_setup [43]: INFO [_provision_components] /opt/seagate/cortx/motr/bin/motr_setup prepare --config yaml:///etc/cortx/cluster.conf --services io
2022-08-18 12:41:21 cortx_setup [43]: INFO [_provision_components] /opt/seagate/cortx/hare/bin/hare_setup prepare --config yaml:///etc/cortx/cluster.conf --services all
2022-08-18 12:41:32 cortx_setup [43]: INFO [_provision_components] /opt/seagate/cortx/utils/bin/utils_setup config --config yaml:///etc/cortx/cluster.conf --services all
2022-08-18 12:41:32 cortx_setup [43]: INFO [_provision_components] /opt/seagate/cortx/motr/bin/motr_setup config --config yaml:///etc/cortx/cluster.conf --services io
2022-08-18 12:41:33 cortx_setup [43]: INFO [_provision_components] /opt/seagate/cortx/hare/bin/hare_setup config --config yaml:///etc/cortx/cluster.conf --services all
2022-08-18 12:41:48 cortx_setup [43]: INFO [_provision_components] /opt/seagate/cortx/utils/bin/utils_setup init --config yaml:///etc/cortx/cluster.conf --services all
2022-08-18 12:41:50 cortx_setup [43]: INFO [_provision_components] /opt/seagate/cortx/motr/bin/motr_setup init --config yaml:///etc/cortx/cluster.conf --services io
2022-08-18 12:41:51 cortx_setup [43]: INFO [_provision_components] /opt/seagate/cortx/hare/bin/hare_setup init --config yaml:///etc/cortx/cluster.conf --services all
2022-08-18 12:42:23 cortx_setup [43]: INFO [cluster_deploy] Finished cluster bootstrap on 7ba8990a85d38508ca9bac4ff516d3ea:cortx-data-g0-0

[root@ssc-vm-rhev4-2619 ~]# kubectl logs -c cortx-setup cortx-ha-f9748b9f-g4zzc
log: cortx_setup config apply -f yaml:///etc/cortx/solution/cluster.yaml -c 'yaml:///etc/cortx/cluster.conf'
2022-08-18 12:40:36 cortx_setup [19]: INFO [config_apply] Applying config yaml:///etc/cortx/solution/cluster.yaml
log: cortx_setup config apply -f yaml:///etc/cortx/solution/config.yaml -c 'yaml:///etc/cortx/cluster.conf'
2022-08-18 12:40:38 cortx_setup [42]: INFO [config_apply] Applying config yaml:///etc/cortx/solution/config.yaml
2022-08-18 12:40:39 cortx_setup [42]: WARNING [save] Release key cortx>common>release>name is missing or has incorrect value.
2022-08-18 12:40:39 cortx_setup [42]: INFO [save] Adding key "cortx>common>release>name" and value "CORTX" in confstore.
2022-08-18 12:40:39 cortx_setup [42]: WARNING [save] Release key cortx>common>release>version is missing or has incorrect value.
2022-08-18 12:40:39 cortx_setup [42]: INFO [save] Adding key "cortx>common>release>version" and value "2.0.0-7581" in confstore.
log: cortx_setup cluster bootstrap -c 'yaml:///etc/cortx/cluster.conf'
2022-08-18 12:41:17 cortx_setup [43]: INFO [_validate_provisioning_status] Recent phase for this node is None and recent status is None. Performing deployment on this node.
2022-08-18 12:41:17 cortx_setup [43]: INFO [cluster_deploy] Starting cluster bootstrap on 7d9ee991ae6ce628a0e10294bdebaa6b:cortx-ha
2022-08-18 12:41:17 cortx_setup [43]: INFO [_provision_components] /opt/seagate/cortx/utils/bin/utils_setup post_install --config yaml:///etc/cortx/cluster.conf --services all
2022-08-18 12:41:19 cortx_setup [43]: INFO [_provision_components] /opt/seagate/cortx/ha/bin/ha_setup post_install --config yaml:///etc/cortx/cluster.conf --services all
2022-08-18 12:41:21 cortx_setup [43]: INFO [_provision_components] /opt/seagate/cortx/utils/bin/utils_setup prepare --config yaml:///etc/cortx/cluster.conf --services all
2022-08-18 12:41:21 cortx_setup [43]: INFO [_provision_components] /opt/seagate/cortx/ha/bin/ha_setup prepare --config yaml:///etc/cortx/cluster.conf --services all
2022-08-18 12:41:22 cortx_setup [43]: INFO [_provision_components] /opt/seagate/cortx/utils/bin/utils_setup config --config yaml:///etc/cortx/cluster.conf --services all
2022-08-18 12:41:23 cortx_setup [43]: INFO [_provision_components] /opt/seagate/cortx/ha/bin/ha_setup config --config yaml:///etc/cortx/cluster.conf --services all
2022-08-18 12:42:24 cortx_setup [43]: INFO [_provision_components] /opt/seagate/cortx/utils/bin/utils_setup init --config yaml:///etc/cortx/cluster.conf --services all
2022-08-18 12:42:25 cortx_setup [43]: INFO [_provision_components] /opt/seagate/cortx/ha/bin/ha_setup init --config yaml:///etc/cortx/cluster.conf --services all
2022-08-18 12:42:26 cortx_setup [43]: INFO [cluster_deploy] Finished cluster bootstrap on 7d9ee991ae6ce628a0e10294bdebaa6b:cortx-ha

[root@ssc-vm-rhev4-2619 ~]# kubectl logs -c cortx-setup cortx-client-0
log: cortx_setup config apply -f yaml:///etc/cortx/solution/cluster.yaml -c 'yaml:///etc/cortx/cluster.conf'
2022-08-18 12:40:30 cortx_setup [19]: INFO [config_apply] Applying config yaml:///etc/cortx/solution/cluster.yaml
log: cortx_setup config apply -f yaml:///etc/cortx/solution/config.yaml -c 'yaml:///etc/cortx/cluster.conf'
2022-08-18 12:40:32 cortx_setup [42]: INFO [config_apply] Applying config yaml:///etc/cortx/solution/config.yaml
2022-08-18 12:40:32 cortx_setup [42]: WARNING [save] Release key cortx>common>release>name is missing or has incorrect value.
2022-08-18 12:40:32 cortx_setup [42]: INFO [save] Adding key "cortx>common>release>name" and value "CORTX" in confstore.
2022-08-18 12:40:32 cortx_setup [42]: WARNING [save] Release key cortx>common>release>version is missing or has incorrect value.
2022-08-18 12:40:32 cortx_setup [42]: INFO [save] Adding key "cortx>common>release>version" and value "2.0.0-7581" in confstore.
log: cortx_setup cluster bootstrap -c 'yaml:///etc/cortx/cluster.conf'
2022-08-18 12:41:17 cortx_setup [43]: INFO [_validate_provisioning_status] Recent phase for this node is None and recent status is None. Performing deployment on this node.
2022-08-18 12:41:17 cortx_setup [43]: INFO [cluster_deploy] Starting cluster bootstrap on c962d1686b3c5f268f5cfe967e45ab6b:cortx-client-0
2022-08-18 12:41:17 cortx_setup [43]: INFO [_provision_components] /opt/seagate/cortx/utils/bin/utils_setup post_install --config yaml:///etc/cortx/cluster.conf --services all
2022-08-18 12:41:19 cortx_setup [43]: INFO [_provision_components] /opt/seagate/cortx/motr/bin/motr_setup post_install --config yaml:///etc/cortx/cluster.conf --services motr_client
2022-08-18 12:41:20 cortx_setup [43]: INFO [_provision_components] /opt/seagate/cortx/hare/bin/hare_setup post_install --config yaml:///etc/cortx/cluster.conf --services all
2022-08-18 12:41:22 cortx_setup [43]: INFO [_provision_components] /opt/seagate/cortx/utils/bin/utils_setup prepare --config yaml:///etc/cortx/cluster.conf --services all
2022-08-18 12:41:23 cortx_setup [43]: INFO [_provision_components] /opt/seagate/cortx/motr/bin/motr_setup prepare --config yaml:///etc/cortx/cluster.conf --services motr_client
2022-08-18 12:41:24 cortx_setup [43]: INFO [_provision_components] /opt/seagate/cortx/hare/bin/hare_setup prepare --config yaml:///etc/cortx/cluster.conf --services all
2022-08-18 12:41:34 cortx_setup [43]: INFO [_provision_components] /opt/seagate/cortx/utils/bin/utils_setup config --config yaml:///etc/cortx/cluster.conf --services all
2022-08-18 12:41:35 cortx_setup [43]: INFO [_provision_components] /opt/seagate/cortx/motr/bin/motr_setup config --config yaml:///etc/cortx/cluster.conf --services motr_client
2022-08-18 12:41:35 cortx_setup [43]: INFO [_provision_components] /opt/seagate/cortx/hare/bin/hare_setup config --config yaml:///etc/cortx/cluster.conf --services all
2022-08-18 12:41:49 cortx_setup [43]: INFO [_provision_components] /opt/seagate/cortx/utils/bin/utils_setup init --config yaml:///etc/cortx/cluster.conf --services all
2022-08-18 12:41:50 cortx_setup [43]: INFO [_provision_components] /opt/seagate/cortx/motr/bin/motr_setup init --config yaml:///etc/cortx/cluster.conf --services motr_client
2022-08-18 12:41:51 cortx_setup [43]: INFO [_provision_components] /opt/seagate/cortx/hare/bin/hare_setup init --config yaml:///etc/cortx/cluster.conf --services all
2022-08-18 12:42:21 cortx_setup [43]: INFO [cluster_deploy] Finished cluster bootstrap on c962d1686b3c5f268f5cfe967e45ab6b:cortx-client-0

@d-nayak d-nayak merged commit 1559de0 into Seagate:main Aug 22, 2022
@pavankrishnat
Copy link
Contributor

@seagate-sarang-sawant

# hctl bootstrap --mkfs singlenode.yaml
2022-08-22 08:32:41: Generating cluster configuration...2022-08-22 08:32:41,400 cfgen 28887 [INFO] Checking dhall versions
2022-08-22 08:32:41,410 cfgen 28887 [INFO] Validating the provided CDF file structure
INFO:cfgen:Validating the provided CDF file structure
2022-08-22 08:32:41,449 cfgen 28887 [INFO] Enriching cluster description
INFO:cfgen:Enriching cluster description
2022-08-22 08:32:42,388 cfgen 28887 [INFO] Validating node descriptions
INFO:cfgen:Validating node descriptions
2022-08-22 08:32:42,389 cfgen 28887 [INFO] Validating pool descriptions
INFO:cfgen:Validating pool descriptions
2022-08-22 08:32:42,389 cfgen 28887 [INFO] Validating profile descriptions
INFO:cfgen:Validating profile descriptions
2022-08-22 08:32:42,390 cfgen 28887 [INFO] Validations passed
INFO:cfgen:Validations passed
2022-08-22 08:32:42,394 cfgen 28887 [INFO] Generating consul configuration
INFO:cfgen:Generating consul configuration
2022-08-22 08:32:42,395 cfgen 28887 [INFO] Generating consul KV structure
INFO:cfgen:Generating consul KV structure
2022-08-22 08:32:42,399 cfgen 28887 [INFO] Generating confd.dhall
INFO:cfgen:Generating confd.dhall
 OK
2022-08-22 08:32:43: Starting Consul server on this node............. OK
2022-08-22 08:32:54: Importing configuration into the KV store... OK
2022-08-22 08:32:54: Starting Consul on other nodes...Consul ready on all nodes
2022-08-22 08:32:54: Updating Consul configuration from the KV store... OK
2022-08-22 08:32:56: Waiting for the RC Leader to get elected.......... OK
2022-08-22 08:33:04: Starting Motr (phase1, mkfs)... OK
2022-08-22 08:33:11: Starting Motr (phase1, m0d)... OK
2022-08-22 08:33:13: Starting Motr (phase2, mkfs)... OK
2022-08-22 08:33:23: Starting Motr (phase2, m0d)... OK
2022-08-22 08:33:26: Checking health of services... OK

Can we remove this duplicate messages, is it expected.

# for free to subscribe to this conversation on GitHub. Already have an account? #.
Reviewers

@mssawant mssawant mssawant approved these changes

@pavankrishnat pavankrishnat pavankrishnat approved these changes

@d-nayak d-nayak d-nayak approved these changes

@Shreya-18 Shreya-18 Awaiting requested review from Shreya-18

Assignees

@mssawant mssawant

@vaibhavparatwar vaibhavparatwar

@Shreya-18 Shreya-18

@d-nayak d-nayak

Labels
cla-signed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@seagate-sarang-sawant @pavankrishnat @mssawant @d-nayak @vaibhavparatwar @Shreya-18