Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

#

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Jump to bottom

Zipkin test failures #862

Closed
ctlong opened this issue Jun 7, 2023 · 1 comment · Fixed by #914
Closed

Zipkin test failures #862

ctlong opened this issue Jun 7, 2023 · 1 comment · Fixed by #914
Assignees
@ctlong

Comments

@ctlong
Copy link
Member

ctlong commented Jun 7, 2023

We've been seeing a lot of flakiness in the CATs zipkin test recently. Particularly when trying to push the spring-sleuth asset, which happens to be ~7 years old.

Summarizing 1 Failure:
  [FAIL] [routing] Zipkin Tracing [BeforeEach] when zipkin tracing is enabled when zipkin headers are not in the request the sleuth error response has no error
  /go/src/github.com/cloudfoundry/cf-acceptance-tests/routing/zipkin_tracing.go:36
App Report 

***************************************************
  ***** APP REPORT: CATS-7-APP-9bfe688507c8088a *****
  ***************************************************

  [2023-06-07 05:08:48.93 (UTC)]> cf app CATS-7-APP-9bfe688507c8088a --guid 
  64db5f4d-a109-4fb2-b052-1990926a309c

  [2023-06-07 05:08:49.93 (UTC)]> cf logs --recent CATS-7-APP-9bfe688507c8088a 
  Retrieving logs for app CATS-7-APP-9bfe688507c8088a in org CATS-7-ORG-d1655112e09a7151 / space CATS-7-SPACE-713a9c3fc1938cde as CATS-7-USER-a9f1e7301fe48695...

     2023-06-07T05:03:52.83+0000 [API/1] OUT Added process: "web"
     2023-06-07T05:03:52.84+0000 [API/1] OUT Created app with guid 64db5f4d-a109-4fb2-b052-1990926a309c
     2023-06-07T05:03:52.86+0000 [API/1] OUT Applied manifest to app with guid 64db5f4d-a109-4fb2-b052-1990926a309c (---
     2023-06-07T05:03:52.87+0000 [API/1] OUT applications:
     2023-06-07T05:03:52.87+0000 [API/1] OUT - name: CATS-7-APP-9bfe688507c8088a
     2023-06-07T05:03:52.87+0000 [API/1] OUT path: "/go/src/github.com/cloudfoundry/cf-acceptance-tests/assets/spring-sleuth/spring-sleuth.jar"
     2023-06-07T05:03:52.87+0000 [API/1] OUT memory: 1024M
     2023-06-07T05:03:52.87+0000 [API/1] OUT default-route: true
     2023-06-07T05:03:52.87+0000 [API/1] OUT buildpacks:
     2023-06-07T05:03:52.87+0000 [API/1] OUT - java_buildpack
     2023-06-07T05:03:52.88+0000 [API/1] OUT )
     2023-06-07T05:03:56.59+0000 [API/1] OUT Uploading app package for app with guid 64db5f4d-a109-4fb2-b052-1990926a309c
     2023-06-07T05:04:05.81+0000 [API/0] OUT Creating build for app with guid 64db5f4d-a109-4fb2-b052-1990926a309c
     2023-06-07T05:04:05.94+0000 [STG/0] OUT Downloading java_buildpack...
     2023-06-07T05:04:05.95+0000 [STG/0] OUT Downloaded java_buildpack
     2023-06-07T05:04:05.95+0000 [STG/0] OUT Cell 3044d081-e428-4159-9dd3-d291523f6c43 creating container for instance 9e378ea2-f762-4f0a-bca7-462e59d7073d
     2023-06-07T05:04:06.61+0000 [STG/0] OUT Security group rules were updated
     2023-06-07T05:04:06.63+0000 [STG/0] OUT Cell 3044d081-e428-4159-9dd3-d291523f6c43 successfully created container for instance 9e378ea2-f762-4f0a-bca7-462e59d7073d
     2023-06-07T05:04:07.32+0000 [STG/0] OUT Downloading app package...
     2023-06-07T05:04:10.67+0000 [STG/0] OUT Downloaded app package (31.6M)
     2023-06-07T05:04:15.79+0000 [STG/0] OUT -----> Java Buildpack v4.59.0 | https://github.com/cloudfoundry/java-buildpack#b890725d
     2023-06-07T05:04:16.18+0000 [STG/0] OUT -----> Downloading Jvmkill Agent 1.17.0_RELEASE from https://java-buildpack.cloudfoundry.org/jvmkill/bionic/x86_64/jvmkill-1.17.0-RELEASE.so (0.1s)
     2023-06-07T05:04:17.19+0000 [STG/0] OUT -----> Downloading Open Jdk JRE 1.8.0_372 from https://java-buildpack.cloudfoundry.org/openjdk/bionic/x86_64/bellsoft-jre8u372%2B7-linux-amd64.tar.gz (0.9s)
     2023-06-07T05:04:19.20+0000 [STG/0] OUT Expanding Open Jdk JRE to .java-buildpack/open_jdk_jre (2.0s)
     2023-06-07T05:04:19.20+0000 [STG/0] OUT JVM DNS caching disabled in lieu of BOSH DNS caching
     2023-06-07T05:04:19.29+0000 [STG/0] OUT -----> Downloading Open JDK Like Memory Calculator 3.13.0_RELEASE from https://java-buildpack.cloudfoundry.org/memory-calculator/bionic/x86_64/memory-calculator-3.13.0-RELEASE.tar.gz (0.0s)
     2023-06-07T05:04:23.46+0000 [STG/0] OUT Loaded Classes: 17173, Threads: 250
     2023-06-07T05:04:23.51+0000 [STG/0] OUT -----> Downloading Client Certificate Mapper 1.11.0_RELEASE from https://java-buildpack.cloudfoundry.org/client-certificate-mapper/client-certificate-mapper-1.11.0-RELEASE.jar (0.0s)
     2023-06-07T05:04:23.57+0000 [STG/0] OUT -----> Downloading Container Security Provider 1.20.0_RELEASE from https://java-buildpack.cloudfoundry.org/container-security-provider/container-security-provider-1.20.0-RELEASE.jar (0.0s)
     2023-06-07T05:04:23.63+0000 [STG/0] OUT -----> Downloading Spring Auto Reconfiguration 2.12.0_RELEASE from https://java-buildpack.cloudfoundry.org/auto-reconfiguration/auto-reconfiguration-2.12.0-RELEASE.jar (0.0s)
     2023-06-07T05:04:23.64+0000 [STG/0] ERR [SpringAutoReconfiguration]      WARN  ATTENTION: The Spring Auto Reconfiguration and shaded Spring Cloud Connectors libraries are being installed. These projects have been deprecated, are no longer receiving updates and should not be used going forward.
     2023-06-07T05:04:23.64+0000 [STG/0] ERR [SpringAutoReconfiguration]      WARN  If you are not using these libraries, set `JBP_CONFIG_SPRING_AUTO_RECONFIGURATION='{enabled: false}'` to disable their installation and clear this warning message. The buildpack will switch its default to disable by default after March 2023. Spring Auto Reconfiguration and its shaded Spring Cloud Connectors will be removed from the buildpack after March 2024.
     2023-06-07T05:04:23.64+0000 [STG/0] ERR [SpringAutoReconfiguration]      WARN  If you are using these libraries, please migrate to java-cfenv immediately. See https://via.vmw.com/EiBW for migration instructions. Once you upgrade this message will go away.
     2023-06-07T05:04:48.01+0000 [STG/0] OUT Exit status 0
     2023-06-07T05:04:48.01+0000 [STG/0] OUT Uploading droplet, build artifacts cache...
     2023-06-07T05:04:48.01+0000 [STG/0] OUT Uploading droplet...
     2023-06-07T05:04:48.01+0000 [STG/0] OUT Uploading build artifacts cache...
     2023-06-07T05:04:48.06+0000 [STG/0] OUT Uploaded build artifacts cache (128B)
     2023-06-07T05:04:50.03+0000 [API/1] OUT Creating droplet for app with guid 64db5f4d-a109-4fb2-b052-1990926a309c
     2023-06-07T05:04:54.49+0000 [STG/0] OUT Uploaded droplet (121.5M)
     2023-06-07T05:04:54.53+0000 [STG/0] OUT Uploading complete
     2023-06-07T05:04:56.40+0000 [STG/0] OUT Cell 3044d081-e428-4159-9dd3-d291523f6c43 stopping instance 9e378ea2-f762-4f0a-bca7-462e59d7073d
     2023-06-07T05:04:56.40+0000 [STG/0] OUT Cell 3044d081-e428-4159-9dd3-d291523f6c43 destroying container for instance 9e378ea2-f762-4f0a-bca7-462e59d7073d
     2023-06-07T05:04:57.62+0000 [API/1] OUT Updated app with guid 64db5f4d-a109-4fb2-b052-1990926a309c ({:droplet_guid=>"1cfc175d-8f4c-4dd4-b1ee-9ceeee6c3668"})
     2023-06-07T05:04:57.65+0000 [API/1] OUT Added process: "task"
     2023-06-07T05:04:57.72+0000 [API/0] OUT Creating revision for app with guid 64db5f4d-a109-4fb2-b052-1990926a309c
     2023-06-07T05:04:57.88+0000 [CELL/0] OUT Cell 94f6d9d9-020f-478f-9426-120ab69e09ad creating container for instance ed0de3c3-25f6-47b9-69b1-0c22
     2023-06-07T05:04:57.96+0000 [API/0] OUT Restarted app with guid 64db5f4d-a109-4fb2-b052-1990926a309c
     2023-06-07T05:04:58.27+0000 [STG/0] OUT Cell 3044d081-e428-4159-9dd3-d291523f6c43 successfully destroyed container for instance 9e378ea2-f762-4f0a-bca7-462e59d7073d
     2023-06-07T05:04:58.62+0000 [CELL/0] OUT Security group rules were updated
     2023-06-07T05:04:58.65+0000 [CELL/0] OUT Cell 94f6d9d9-020f-478f-9426-120ab69e09ad successfully created container for instance ed0de3c3-25f6-47b9-69b1-0c22
     2023-06-07T05:04:58.99+0000 [CELL/0] OUT Downloading droplet...
     2023-06-07T05:05:05.46+0000 [CELL/0] OUT Downloaded droplet (121.5M)
     2023-06-07T05:05:05.46+0000 [CELL/0] OUT Starting health monitoring of container
     2023-06-07T05:05:06.66+0000 [APP/PROC/WEB/0] OUT JVM Memory Configuration: -Xmx419387K -Xss1M -XX:ReservedCodeCacheSize=240M -XX:MaxDirectMemorySize=10M -XX:MaxMetaspaceSize=117188K
     2023-06-07T05:05:40.85+0000 [APP/PROC/WEB/0] OUT 2023-06-07 05:05:40.714  INFO [bootstrap,,,] 8 --- [           main] pertySourceApplicationContextInitializer : 'cloud' property source added
     2023-06-07T05:05:40.93+0000 [APP/PROC/WEB/0] OUT 2023-06-07 05:05:40.876  INFO [bootstrap,,,] 8 --- [           main] nfigurationApplicationContextInitializer : Reconfiguration enabled
     2023-06-07T05:05:43.90+0000 [APP/PROC/WEB/0] OUT 2023-06-07 05:05:43.837  INFO [bootstrap,,,] 8 --- [           main] s.c.a.AnnotationConfigApplicationContext : Refreshing org.springframework.context.annotation.AnnotationConfigApplicationContext@1ee0005: startup date [Wed Jun 07 05:05:43 UTC 2023]; root of context hierarchy
     2023-06-07T05:05:49.14+0000 [APP/PROC/WEB/0] OUT 2023-06-07 05:05:49.140  INFO [bootstrap,,,] 8 --- [           main] f.a.AutowiredAnnotationBeanPostProcessor : JSR-330 'javax.inject.Inject' annotation found and supported for autowiring
     2023-06-07T05:05:49.24+0000 [APP/PROC/WEB/0] OUT 2023-06-07 05:05:49.241  INFO [bootstrap,,,] 8 --- [           main] trationDelegate$BeanPostProcessorChecker : Bean 'configurationPropertiesRebinderAutoConfiguration' of type [class org.springframework.cloud.autoconfigure.ConfigurationPropertiesRebinderAutoConfiguration$$EnhancerBySpringCGLIB$$e5cae187] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
     2023-06-07T05:05:50.63+0000 [APP/PROC/WEB/0] OUT .   ____          _            __ _ _
     2023-06-07T05:05:50.63+0000 [APP/PROC/WEB/0] OUT /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
     2023-06-07T05:05:50.63+0000 [APP/PROC/WEB/0] OUT ( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
     2023-06-07T05:05:50.63+0000 [APP/PROC/WEB/0] OUT \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
     2023-06-07T05:05:50.63+0000 [APP/PROC/WEB/0] OUT '  |____| .__|_| |_|_| |_\__, | / / / /
     2023-06-07T05:05:50.63+0000 [APP/PROC/WEB/0] OUT =========|_|==============|___/=/_/_/_/
     2023-06-07T05:05:50.63+0000 [APP/PROC/WEB/0] OUT :: Spring Boot ::        (v1.4.0.RELEASE)
     2023-06-07T05:05:51.32+0000 [APP/PROC/WEB/0] OUT 2023-06-07 05:05:51.308  INFO [,,,] 8 --- [           main] c.c.c.ConfigServicePropertySourceLocator : Fetching config from server at: http://localhost:8888
     2023-06-07T05:05:51.69+0000 [APP/PROC/WEB/0] OUT 2023-06-07 05:05:51.692  WARN [,,,] 8 --- [           main] c.c.c.ConfigServicePropertySourceLocator : Could not locate PropertySource: I/O error on GET request for "http://localhost:8888/application/cloud": Connection refused (Connection refused); nested exception is java.net.ConnectException: Connection refused (Connection refused)
     2023-06-07T05:05:51.70+0000 [APP/PROC/WEB/0] OUT 2023-06-07 05:05:51.701  INFO [,,,] 8 --- [           main] pertySourceApplicationContextInitializer : 'cloud' property source added
     2023-06-07T05:05:51.70+0000 [APP/PROC/WEB/0] OUT 2023-06-07 05:05:51.702  INFO [,,,] 8 --- [           main] nfigurationApplicationContextInitializer : Reconfiguration enabled
     2023-06-07T05:05:51.70+0000 [APP/PROC/WEB/0] OUT 2023-06-07 05:05:51.704  INFO [,,,] 8 --- [           main] hello.Application                        : The following profiles are active: cloud
     2023-06-07T05:05:51.76+0000 [APP/PROC/WEB/0] OUT 2023-06-07 05:05:51.758  INFO [,,,] 8 --- [           main] ationConfigEmbeddedWebApplicationContext : Refreshing org.springframework.boot.context.embedded.AnnotationConfigEmbeddedWebApplicationContext@6be46e8f: startup date [Wed Jun 07 05:05:51 UTC 2023]; parent: org.springframework.context.annotation.AnnotationConfigApplicationContext@1ee0005
     2023-06-07T05:06:04.92+0000 [APP/PROC/WEB/0] OUT 2023-06-07 05:06:04.924  WARN [,,,] 8 --- [           main] o.s.c.a.ConfigurationClassPostProcessor  : Cannot enhance @Configuration bean definition 'refreshScope' since its singleton instance has been created too early. The typical cause is a non-static @Bean method with a BeanDefinitionRegistryPostProcessor return type: Consider declaring such methods as 'static'.
     2023-06-07T05:06:05.66+0000 [APP/PROC/WEB/0] OUT 2023-06-07 05:06:05.667  INFO [,,,] 8 --- [           main] o.s.cloud.context.scope.GenericScope     : BeanFactory id=8a5f9aa8-7bcf-3eae-a795-32417afe47a6
     2023-06-07T05:06:05.74+0000 [APP/PROC/WEB/0] OUT 2023-06-07 05:06:05.747  INFO [,,,] 8 --- [           main] f.a.AutowiredAnnotationBeanPostProcessor : JSR-330 'javax.inject.Inject' annotation found and supported for autowiring
     2023-06-07T05:06:05.98+0000 [APP/PROC/WEB/0] OUT 2023-06-07 05:06:05.986  INFO [,,,] 8 --- [           main] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.cloud.sleuth.instrument.async.AsyncDefaultAutoConfiguration$DefaultAsyncConfigurerSupport' of type [class org.springframework.cloud.sleuth.instrument.async.AsyncDefaultAutoConfiguration$DefaultAsyncConfigurerSupport$$EnhancerBySpringCGLIB$$6f27c4d] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
     2023-06-07T05:06:07.06+0000 [APP/PROC/WEB/0] OUT 2023-06-07 05:06:07.062  INFO [,,,] 8 --- [           main] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.cloud.autoconfigure.ConfigurationPropertiesRebinderAutoConfiguration' of type [class org.springframework.cloud.autoconfigure.ConfigurationPropertiesRebinderAutoConfiguration$$EnhancerBySpringCGLIB$$e5cae187] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
     2023-06-07T05:06:07.13+0000 [HEALTH/0] ERR Timed out after 1m0s (30 attempts) waiting for readiness check to succeed: failed to make TCP connection to 10.255.16.133:8080: dial tcp 10.255.16.133:8080: connect: connection refused
     2023-06-07T05:06:07.13+0000 [CELL/0] ERR Failed after 1m1.665s: readiness health check never passed.

@ctlong
Copy link
Member Author

ctlong commented Jun 7, 2023

Started CF Slack conversations in cats and in cf-for-vms-networking.

@ctlong ctlong self-assigned this Jun 12, 2023
@ctlong ctlong mentioned this issue Aug 9, 2023
Merged
7 tasks
@ctlong ctlong linked a pull request Aug 9, 2023 that will close this issue
fix(zipkin): increase app readiness timeout to reduce test flakes #914
Merged
7 tasks
@ctlong ctlong removed a link to a pull request Aug 9, 2023
fix(zipkin): increase app readiness timeout to reduce test flakes #914
Merged
7 tasks
@ctlong ctlong linked a pull request Aug 9, 2023 that will close this issue
fix(zipkin): increase app readiness timeout to reduce test flakes #914
Merged
7 tasks
@ctlong ctlong closed this as completed Aug 21, 2023
# for free to join this conversation on GitHub. Already have an account? # to comment
Assignees

@ctlong ctlong

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix(zipkin): increase app readiness timeout to reduce test flakes cloudfoundry/cf-acceptance-tests
1 participant
@ctlong