Initial Botsing experiments with Atos Supersede IF component. No crash replicating tests are generated.

[jesus-gorronogoitia]

Characteristics

• Issue Type: test report
• Reproducibility: always
• Severity: N/A
• Tool/Service/Component: Botsing 1.0.0
• Execution Environment: CentOS7
• Reporter: Jesús Gorroñogoitia jesus.gorronogoitia@atos.net

Description

We have initially try to run Botsing to analyse and generate crash replicating tests for Atos Supersede IF component.
Initially we use the stack traces we found in our production server. As initial examples we selected the stack traces: exception14.log and exceptiong16.log located here.

We run Botsing using the scripts run_botsing_exception14.sh and run_botsing_exception16.sh located here. We run dspot with the target frame set to 2. Botsing, after 30 min, did not find any test.

So, We decided to create manually a crash replicating tests for both cases.
For exception14 the crash replicating test is:

         String testUserName = "test";

	@Test //(expected = org.wso2.carbon.user.core.UserStoreException.class)
	public void throwUserExistWhenAddUserTest() throws UserStoreException, MalformedURLException{
		IFAuthenticationManager am = new IFAuthenticationManager();
                User user = am.getUser(testUserName);
		try {
			//If user does not exist, create user
			if (user==null){
				user = createTestUser();
		    	am.addUser(user, testUserPassword, requirePasswordChange);
			}
			//Try to create a new user
			am.addUser(user, testUserPassword, requirePasswordChange);
		} catch (Exception e) {
			e.printStackTrace();
			throw e;
		} finally {
			am.deleteUser(user);
		}
	}

	private User createTestUser() throws UserStoreException {
		return createTestUser(testUserName);
	}
	
	private User createTestUser(String userName) throws UserStoreException {
		User user = new User();
		user.setUserName(userName);
		user.setFirstname("User Test firstname");
		user.setLastname("User Test lastname");
		user.setOrganization("User Test organization");
		user.setAddress("User Test address");
		user.setCountry("User Test country");
		user.setEmail("user@organization.org");
		user.setTelephone("800-555-55-55");
		user.setMobile("800-555-55-55");
		user.setIm("User Test IM");
    	try {
			user.setUrl(new URL("http://organization.org"));
		} catch (MalformedURLException e) {
			// TODO Auto-generated catch block
			e.printStackTrace();
		}
    	
    	//Adding roles
    	Set<Role>roles = new HashSet<Role>();
    	Set<Role> allRoles = am.getAllRoles();
    	Role role = createTestRole();
    	if (!allRoles.contains(role)){
    		am.addRole(role);
    	}
//    	for (Role role: allRoles){
//    		if (role.getRoleName().contains("testRole")){
//    			roles.add(role);
//    		}
//    	}
    	roles.add(role);
    	user.setRoles(roles);
		return user;
	}

For exception16 the test is:

	@Test (expected=org.wso2.carbon.user.core.UserStoreException.class)
	public void throwUserDoesNotExistWhenDeleteUserTest() throws UserStoreException, MalformedURLException{
		try {
                        IFAuthenticationManager am = new IFAuthenticationManager();
			User user = createTestUser("userdoesnotexist");
			am.deleteUser(user);
		} catch (Exception e) {
			e.printStackTrace();
			throw e;
		}
	}

When running these tests we realized that the thrown exceptions were not exactly the ones reported in production server. The latter was included within the stack trace of the former within the section "Caused by: ...", So we created new exception14b.log and exception16b.log in the same folder (see above) that contained the header frames in the stack traces triggered when running the tests. And we try Botsing again with similar negative results.
Indeed, in these cases "b", if we run Botsing with target frame = 3, we get the following exception:

Exception in thread "main" java.lang.ExceptionInInitializerError
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:348)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.evosuite.testcase.statements.MethodStatement$1.execute(MethodStatement.java:257)
	at org.evosuite.testcase.statements.AbstractStatement.exceptionHandler(AbstractStatement.java:169)
	at org.evosuite.testcase.statements.MethodStatement.execute(MethodStatement.java:220)
	at org.evosuite.testcase.execution.TestRunnable.executeStatements(TestRunnable.java:307)
	at org.evosuite.testcase.execution.TestRunnable.call(TestRunnable.java:213)
	at org.evosuite.testcase.execution.TestRunnable.call(TestRunnable.java:55)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.SecurityException: Security manager blocks ("javax.net.ssl.SSLPermission" "setHostnameVerifier")
java.lang.Thread.getStackTrace(Thread.java:1559)
org.evosuite.runtime.sandbox.MSecurityManager.checkPermission(MSecurityManager.java:434)
javax.net.ssl.HttpsURLConnection.setDefaultHostnameVerifier(HttpsURLConnection.java:233)
eu.supersede.integration.api.security.IFAuthenticationManager.<clinit>(IFAuthenticationManager.java)
java.lang.Class.forName0(Native Method)
java.lang.Class.forName(Class.java:348)
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(Method.java:498)
org.evosuite.testcase.statements.MethodStatement$1.execute(MethodStatement.java:257)
org.evosuite.testcase.statements.AbstractStatement.exceptionHandler(AbstractStatement.java:169)
org.evosuite.testcase.statements.MethodStatement.execute(MethodStatement.java:220)
org.evosuite.testcase.execution.TestRunnable.executeStatements(TestRunnable.java:307)
org.evosuite.testcase.execution.TestRunnable.call(TestRunnable.java:213)
org.evosuite.testcase.execution.TestRunnable.call(TestRunnable.java:55)
java.util.concurrent.FutureTask.run(FutureTask.java:266)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)

	at org.evosuite.runtime.sandbox.MSecurityManager.checkPermission(MSecurityManager.java:452)
	at javax.net.ssl.HttpsURLConnection.setDefaultHostnameVerifier(HttpsURLConnection.java:233)
	at eu.supersede.integration.api.security.IFAuthenticationManager.<clinit>(IFAuthenticationManager.java)
	... 16 more

Steps to reproduce

See description section

Other files and URLs

See all required artifacts to run the experiment in this folder

Are we using Botsing correctly? Are these examples good candidates for Botsing? What else can we do to get better results.
Please, provide us feedback as soon as you can. Thank you.

Relationships

Help on issue template

Preview to follow the link or open file .github/ISSUE_DOC.md

[pderakhshanfar]

Dear @jesus-gorronogoitia ,

According to the exception that you got from the botsing, I can see that the problem is in the security manager.

What is Java Security Manager? If the generated tests interact with their environment, the execution of them the during the search may lead to unexpected or undesirable side-effects. For instance, they may unexpectedly remove some files from the system.

Botsing uses this security manager to avoid these unexpected side effects. However, this limits some interactions for the generated test cases. You can see these limitations in this paper from Evosuite. In this paper, we can see that All java.net.SocketPermissions are strictly denied.

[jesus-gorronogoitia]

Hi @pderakhshanfar ,
Your feedback is about the exception when the target frame is 3. Sincerely, you are suggesting me to read your paper? Cannot you provide me some hints, before I read the paper?. What is supposed I should do to get Botsing working?
And what about the other issues? Why Botsing is not capable to find crash replicating tests for the other experiments and target frames? Can I modify Botsing configuration to improve results? Are these exceptions (and associated manual replicating tests) quite complex for Botsing?
Please, comment any strategy that come to your mind that you think could help.
Think, that we, as end users, are not suppose to read a scientific paper to get a tool work. I will do it, as you suggest, but not now, I don't have time, and you should not expect most of your potential end users would do it.
Hope you understand this.
Thanks for your support.

[pderakhshanfar]

@jesus-gorronogoitia No, I just wanted to attach the paper too. However, the answer was already mentioned in my previous message. To avoid unexpected events during the execution of the generated tests, we have a security manager which put some limitations on the classes that we can test (here, the class which is pointed from target frame 3). So, these kinds of classes cannot be tested bye Botsing or EvoSuite. For the other target frames, I can see that for some reason Botsing cannot generate the test for replicating the given crash. According to our large-scale evaluation, there are still lots of challenges remains. So, there are still some cases that we can not reproduce. In your case, I cannot say which challenges happened here, because I should do a manual analysis of the source code of the software under the test.

[jesus-gorronogoitia]

Hi @pderakhshanfar Thanks for clarifications.
So, target frame 3 is not possible, understood. So, for success in Botsing test generation, is there something we can do? If not, I will try with other exceptions, but the remaining ones are even more complex.
Consider that by this month in D5.1 we have to report KPIs related to Botsing. IMHO, it is not a risk to report that Botsing is not able to reproduce our runtime exceptions (Botsing is not supposed to do magic with all possible industrial exceptions), but we all (including Botsing team) should justify the reasons.
If you think that is worthy to try with other Botsing configurations (increasing time budget, etc), let me know.

[pderakhshanfar]

For this moment, no. However, we are adding new features that will be available soon. With these new features, Botsing will have more rate of crash reproduction. We are working on them to release them as soon as possible.

[jesus-gorronogoitia]

Thanks @pderakhshanfar keep me informed of incoming Botsing releases

[jesus-gorronogoitia]

Hi @pderakhshanfar Thinking on the Botsing (Evocrash) security manager restrictions, and after browsing the paper you referred to, I wonder if some of these restrictions could not be disable by the user configuration. Otherwise, I am afraid Botsing wouldn't be able to reproduce any of your crash exceptions, since they all involved network proxies that mediates between services in a microservice architecture. With this security restrictions, it seems Botsing cannot work with network communications, and therefore, it won't work with our use case. Is there any way to relax these restrictions?

[pderakhshanfar]

Hi @jesus-gorronogoitia Yes you can disable the security manager by passing this command while you are calling the botsing: -Dsandbox=False
However, disabling it can be risky. It may lead to removing some of the critical files of OS.

[jesus-gorronogoitia]

Hi @pderakhshanfar I do not refer to remove all the security manager restrictions but only those that refer to java.net.SocketPermissions. In other words, could be possible to configure the restrictions on the security manager?

[pderakhshanfar]

Dear @jesus-gorronogoitia I am afraid there is no easy way to disable only some of the restrictions of sandbox in Evosuite. Also, since we are using Evosuite as a dependency, the only way to achieve this goal is re-implementing all of the modules related to the test executor and sandbox which are already developed by EvoSuite developers.

[jesus-gorronogoitia]

Hi @pderakhshanfar
I've resumed this issue about using Botsing for Atos Supersede IF crash reproduction. In order to run Botsing with security manager disabled, I've run it within a Docker container.
I've invoked Botsing over a couple of crash exceptions (namely 14 and 16 mentioned above) using this command:
java -jar botsing-reproduction-1.0.6.jar -Dsandbox=False -Dsearch_budget=180 -Dpopulation=100 -Dtest_dir=crash-reproduction-tests -crash_log ./logs/exception14b.log -target_frame 3 -project_cp ./libs

with security manager disabled by -Dsandbox=False

and ./logs/exception14b.log showing:

org.wso2.carbon.user.core.UserStoreException: Username 'test' already exists in the system. Please pick another username.
        at eu.supersede.integration.api.security.IFUserStoreManager.handleException(IFUserStoreManager.java:573)
        at eu.supersede.integration.api.security.IFUserStoreManager.addUser(IFUserStoreManager.java:142)
        at eu.supersede.integration.api.security.IFAuthenticationManager.addUser(IFAuthenticationManager.java:165)

Botsing execution ends after budget with no one single test generated:

...
2:53:13.304 [main] INFO  e.s.b.g.s.GuidedGeneticAlgorithm - Best fitness in the current population: 4.5 | 17000
12:53:13.944 [main] INFO  e.s.b.g.s.GuidedGeneticAlgorithm - Best fitness in the current population: 4.5 | 17100
12:53:14.082 [main] INFO  e.s.b.g.s.GuidedGeneticAlgorithm - MaxTime :                        181 / 180          Finished!
12:53:14.083 [main] INFO  e.s.b.g.s.GuidedGeneticAlgorithm - Best fitness in the current population: 4.5 | 17200
12:53:14.083 [main] INFO  e.s.b.g.s.GuidedGeneticAlgorithm - MaxTime :                        181 / 180          Finished!
12:53:14.083 [main] INFO  e.s.b.g.s.GuidedGeneticAlgorithm - The search process is finished.
12:53:14.083 [main] INFO  e.s.b.g.s.GuidedGeneticAlgorithm - Best fitness in the final population is: 4.5. PT: 8 seconds
12:53:14.084 [main] INFO  e.s.b.t.s.BotsingIndividualStrategy - * The target crash is not covered! The best solution has 4.5 fitness value.
12:53:14.084 [main] INFO  e.s.b.t.s.BotsingIndividualStrategy - The best test is:(non-minimized version:
)
12:53:14.088 [main] INFO  e.s.b.reproduction.CrashReproduction - * Minimizing test suite
12:53:14.091 [main] INFO  e.s.b.reproduction.CrashReproduction - * Compiling and checking tests
12:53:14.104 [main] INFO  e.s.b.reproduction.CrashReproduction - * Writing JUnit test case 'IFAuthenticationManager_ESTest' to crash-reproduction-tests
12:53:14.133 [main] INFO  e.s.b.reproduction.CrashReproduction - The solution test is saved!

A possible hand-crafted test case for this exception 14 is shown above.

So, I can't make Botsing to generate any crash replicating test despite I disabled the security manager.
Is there any way to debug Botsing execution (and get detailed logs) to check what results Bolsing is obtaining when running the tests it generates, to be sure they are getting access to the external services that eu.supersede.integration.api.security.IFAuthenticationManager.addUser is invoking?

Please, let me know any means to analyze Botsing execution, so we can understand the reasons of Botsing not generating test-replicating test

[jesus-gorronogoitia]

Hi @pderakhshanfar
I detected that there was not network access within my Docker container sandbox executing Botsing, so I enabled it. After that, I executed within the container a handcrafted test that reproduced the exception 16 (test reported above).
However, executing Botsing again, within the container with network connection, and with security manager disabled, no crash replicating test is generated after exhausting the 30 min budget.
Botsing was executed with the same configuration reported in previous comment. Same result output was produced.
Therefore, if order to get progress I need instructions to debug Botsing execution, so I can provide you those logs, or for experimenting with other configuration.
Thank you so much for your support.

[gibello]

Hi @pderakhshanfar ,
What about disabling security manager in a Docker container ?
It should not be critical system files that can be affected... well, it might be, but only the container should be affected, not the host ?
Or did I miss something ?

[pderakhshanfar]

@jesus-gorronogoitia Botsing cannot reproduce all of the crashes. It reproduces a crash if it reaches to fitness function 0.0. According to the log that you have sent, the best fitness value is 4.5 in your case. Due to the challenging nature of the crash reproduction problem, there are lots of reasons for Botsing to be unsuccessful in reproducing a crash. If it cannot reproduce your crash even by setting the sandbox off, it means that botsing faces other challenges here. We have detected these challenges in our previous studies, and we are continually trying to improve the reproduction rate by addressing those challenges. However, it does not guarantee that Botsing will reproduce all of the crashes in the next releases.
Due to this limitation, I am suggesting that you apply Botsing on different crashes and different frame levels.

For instance, in order to reproduce crashes for XWiki, we collected more than 70 crashes from their JIRA and ran Botsing on their frames for multiple times. Next, we collected the crashes that botsing could reproduce them. We automated this process by ExRunner.

[pderakhshanfar]

Hi @pderakhshanfar ,
What about disabling security manager in a Docker container ?
It should not be critical system files that can be affected... well, it might be, but only the container should be affected, not the host ?
Or did I miss something ?

Hi @gibello,

Botsing generates and executes lots of random tests during the search. These generated tests may remove or change some critical files. However, I think, if we run Botsing on a docker container, the generated tests should not have access to the host files. However, I have always run Botsing and EvoSuite with enabled sandbox, and I do not have experience in running these tools on docker with disabled sandbox.

[jesus-gorronogoitia]

Hi @pderakhshanfar , @xdevroey
I've conducted new Botsing experiments feeding it with behavioural model seeding, but still I am getting no crash replicating tests, keeping the best fitness value in 4.5.
I will keep experimenting with other frames and crashes. Do you recommend me to increase the budget or to tweak another Botsing parameter? Will it give me any chance?

[pderakhshanfar]

Hi @jesus-gorronogoitia ,

Model seeding only drives the search process to use the behavior of classes which are observed in the source code and/or manually written test cases. If the observed behavior does not contain any helpful content (e.g., call sequences close to the behavior, which leads to crash reproduction), it cannot improve crash reproduction. According to our experiment on botsing model seeding, we witnessed that model seeding can be helpful for only some specific crashes. So, I recommend you to try botsing model seeding on multiple reported crashes. Changing the parameters does not help usually.
However, if you do not have any other crashes and want to give more chance to botsing for reproducing the existing crashes, You can run botsing with the various seeding probabilities by the following parameter: -Dp_object_pool=<probability_of_seeding>

[jesus-gorronogoitia]

Hi @pderakhshanfar
I've selected the seeding probability to 1.0 in my experiments (I assume the highest). Do you mean I should try them in the range (0,1) to see how different probabilities may impact Botsing search? Shouldn't be the highest (1.0) the one that has most positive influence in Botsing search? I mean, will it help to use lower probabilities?

[jesus-gorronogoitia]

BTW, Is there any way to check in logs that Botsing has disabled the SecurityManager when the flag -Dsandbox=False is passed? I am asking because it seems EvoSuite is not disabling it. See #92

[pderakhshanfar]

Hi @pderakhshanfar
I've selected the seeding probability to 1.0 in my experiments (I assume the highest). Do you mean I should try them in the range (0,1) to see how different probabilities may impact Botsing search? Shouldn't be the highest (1.0) the one that has most positive influence in Botsing search? I mean, will it help to use lower probabilities?

The best probability of seeding is varied according to the case. If the model is small, having high probability may misguide the search process and vice versa.