Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

[CI] Run unit and functional tests separately in CI #368

Merged
merged 7 commits into from
Jul 17, 2023

Conversation

timkimadobe
Copy link
Contributor

@timkimadobe timkimadobe commented Jul 14, 2023

Description

This PR:

  1. Splits unit and functional test steps in the makefile to prevent the occasional interleaving of unit test build process with functional test case execution

Related Issue

Motivation and Context

For future reference, the old method required extracting the true scheme, with output like (note the duplicate AEPEdge scheme differentiated with parentheses):

List of available shared Schemes in xcworkspace
xcodebuild -workspace AEPEdge.xcworkspace -list
Command line invocation:
    /Applications/Xcode-14.1.app/Contents/Developer/usr/bin/xcodebuild -workspace AEPEdge.xcworkspace -list

User defaults from command line:
    IDEPackageSupportUseBuiltinSCM = YES

Information about workspace "AEPEdge":
    Schemes:
        AEPAssurance
        AEPCore
        AEPEdge (AEPEdge project)
        AEPEdge (Pods project)
        AEPEdgeConsent
        AEPEdgeIdentity
        AEPEdgeXCF
        AEPRulesEngine
        AEPServices
        FunctionalTests
        Pods-AEPEdge
        Pods-FunctionalTests
        Pods-TestAppiOS
        Pods-TestApptvOS
        Pods-UnitTests
        Pods-UpstreamIntegrationTests
        SwiftLint
        TestAppiOS
        TestApptvOS
        UnitTests
        UpstreamIntegrationTests

final_scheme=""; \
	if xcodebuild -workspace AEPEdge.xcworkspace -list | grep -q "(AEPEdge project)"; \
	then \
	   final_scheme="AEPEdge (AEPEdge project)" ; \
	   echo $final_scheme ; \
	else \
	   final_scheme="AEPEdge" ; \
	   echo $final_scheme ; \
	fi; \
	xcodebuild test -workspace AEPEdge.xcworkspace -scheme "$final_scheme" -destination 'platform=iOS Simulator,name=iPhone 14' -derivedDataPath build/out -resultBundlePath iosresults.xcresult -enableCodeCoverage YES ADB_SKIP_LINT=YES
AEPEdge (AEPEdge project)
Command line invocation:
    /Applications/Xcode-14.1.app/Contents/Developer/usr/bin/xcodebuild test -workspace AEPEdge.xcworkspace -scheme "AEPEdge (AEPEdge project)" -destination "platform=iOS Simulator,name=iPhone 14" -derivedDataPath build/out -resultBundlePath iosresults.xcresult -enableCodeCoverage YES ADB_SKIP_LINT=YES

How Has This Been Tested?

Screenshots (if appropriate):

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

  • I have signed the Adobe Open Source CLA.
  • My code follows the code style of this project.
  • My change requires a change to the documentation.
  • I have updated the documentation accordingly.
  • I have read the CONTRIBUTING document.
  • I have added tests to cover my changes.
  • All new and existing tests passed.

Update CircleCI script accordingly
@timkimadobe timkimadobe changed the title Build process fix CI test process fix Jul 14, 2023
@timkimadobe timkimadobe changed the title CI test process fix CI testing workflow fix Jul 14, 2023
@codecov
Copy link

codecov bot commented Jul 15, 2023

Codecov Report

Merging #368 (61b1076) into dev (1cf1b14) will increase coverage by 0.05%.
The diff coverage is 100.00%.

❗ Current head 61b1076 differs from pull request most recent head bff3336. Consider uploading reports for the commit bff3336 to get more accurate results

@@            Coverage Diff             @@
##              dev     #368      +/-   ##
==========================================
+ Coverage   96.68%   96.73%   +0.05%     
==========================================
  Files          27       27              
  Lines        1626     1653      +27     
==========================================
+ Hits         1572     1599      +27     
  Misses         54       54              

@timkimadobe timkimadobe mentioned this pull request Jul 15, 2023
10 tasks
Copy link
Contributor Author

@timkimadobe timkimadobe left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the review @emdobrin! Reordered unit and functional tests

@timkimadobe timkimadobe requested a review from emdobrin July 15, 2023 00:15
Copy link
Contributor

@kevinlind kevinlind left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This change looks good but it appears there are still test failures in CI. Are those failures expected?

@kevinlind
Copy link
Contributor

In the CircleCI configuration, I'd recommend running the unit and functional tests in separate steps, instead of just one step which runs both sequentially. The reason is when a failure occurs it is hard to know if the failure is from the unit or functional tests. For example, in this failure the last few lines displayed show a functional test failure but searching through the entire log shows a unit test failed as well.
https://app.circleci.com/pipelines/github/adobe/aepsdk-edge-ios/1094/workflows/8f8abbf7-815b-4877-88ec-2e70a4af49b8/jobs/1745

@kevinlind kevinlind changed the title CI testing workflow fix [CI] Run unit and functional tests separately in CI Jul 17, 2023
@kevinlind kevinlind merged commit f114e14 into adobe:dev Jul 17, 2023
@timkimadobe timkimadobe deleted the build-process-fix branch July 18, 2023 23:15
# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants