Enabling Mutant Generation for Open- and Closed-Source Android Apps

This project was created by the Software Engineering Maintenance and Evolution Research Unit (SEMERU) at the College of William & Mary, in collaboration with The Software Design Lab at the Universidad de los Andes, The University of Sannio, REVEAL at the The University of Lugano, and the Miami University, Oxford, OH. The major goal of the project is to help mobile application developers and testers improve the quality of their test suites. This project empirically derived a set of 38 Android-specific mutation operators which can be automatically seeded into a target application and analyzed on a set of test cases.

Tools

Visit webpage » View code »

Major Wrapper

Overview » View code »

Pitest Wrapper

Overview » View code »

Fault Taxonomy

The black rectangle in the bottom-right part of Figure 1 reports the number of documents tagged as false positive or as un- clear. e other rectangles—marked with the Android and/or with the Java logo—represent the 14 high-level categories that we identified. Categories marked with the Android logo (e.g., Activities and Intents) group together Android-specific bugs while those marked with the Java logo (e.g., Collections and Strings) group bugs that could affect any Java application. Both symbols together indicate categories featuring both Android-specific and Java-related bugs (see e.g., I/O).

Mutant Operators

These operators were implemented in Java (for source code-based mutations in MDroid+) and SMALI (for APK-based mutations in MutAPK). The locations for the mutations are identified by using a Potential Failure Profile (PFP). The PFP lists code locations that could be modified to inject a mutation. The locations of the analyzed apps which can be source code statements, XML tags or locations in other resource files that can be the source of a potential fault, given the faults catalog from the taxonomy.

Available in MDroid+

Available in MutAPK

AST
Changes are done directly over AST

Text
Changes are done over resource files

Category 1 - Activity/Intents

ActivityNotDefined

Text

Description:

Delete an activity < android:name="Activity" /> entry in the Manifest file

How to find a mutant of this kind:

Provide a sequence of events that visits the removed activity

Source Code level - Code Example APK level - Code Example

Before:

<activity android:name=".ImportActivity" android:label="@string/title_import" />

After:

Before:

<activity android:name=".ImportActivity" android:label="@string/title_import" />

After:

DifferentActivityIntentDefinition

AST

Description:

Replace the Activity.class argument in an Intent instantiation

How to find a mutant of this kind:

Provide a sequence of events that exercises the modified intent

Source Code level - Code Example APK level - Code Example

Before:

Intent intent = new Intent(main.this, ImportActivity.class);

After:

Intent intent = new Intent(main.this, ExportActivity.class);

Before:

const-class v3, Lcom/fsck/k9/activity/MessageCompose;  
invoke-direct {v1, v2, v3}, Landroid/content/Intent;-><init>(Landroid/content/Context;Ljava/lang/Class;)V

After:

const-class v1, Lcom/fsck/k9/activity/Accounts; 
invoke-direct {v1, v2, v3}, Landroid/content/Intent;-><init>(Landroid/content/Context;Ljava/lang/Class;)V

InvalidActivityName

Text

Description:

Randomly insert typos in the path of an activity defined in the Manifest file

How to find a mutant of this kind:

Provide a sequence of events that visits the modified activity

Source Code level - Code Example APK level - Code Example

Before:

<activity android:name=".AboutActivity" android:label="@string/title_about" />

After:

<activity android:name=".AbutActivity" android:label="@string/title_about" />

Before:

<activity android:name=".AboutActivity" android:label="@string/title_about" />

After:

<activity android:name=".AbutActivity" android:label="@string/title_about" />

InvalidKeyIntentPutExtra

AST

Description:

Randomly generate a different key in an Intent.putExtra(key, value) call

How to find a mutant of this kind:

Provide a sequence of events that exercises the modified intent

Source Code level - Code Example APK level - Code Example

Before:

intent.putExtra(key, value);

After:

intent.putExtra(“ecab6839856b426fbdae3e6e8c46c38c”, value);

Before:

const-string v1, "message_decryption_result" 
invoke-virtual {v0, v1, p3}, Landroid/content/Intent;->putExtra(Ljava/lang/String;Landroid/os/Parcelable;)Landroid/content/Intent;

After:

const-string v1, "3a55422d7ac44b6aac475cc5122b992b" 
invoke-virtual {v0, v1, p3}, Landroid/content/Intent;->putExtra(Ljava/lang/String;Landroid/os/Parcelable;)Landroid/content/Intent;

InvalidLabel

Text

Description:

Replace the attribute "android:label" in the Manifest file with a random string

How to find a mutant of this kind:

Provide a sequence of events that visits the modified activity and checks for the activity name

Source Code level - Code Example APK level - Code Example

Before:

<activity android:name=".VehicleActivity" android:label="@string/title_vehicle" />

After:

<activity android:name=".VehicleActivity" android:label="ecab6839856b426fbdae3e6e8c46c38c" />

Before:

<activity android:name=".VehicleActivity" android:label="@string/title_vehicle" />

After:

<activity android:name=".VehicleActivity" android:label="ecab6839856b426fbdae3e6e8c46c38c" />

NullIntent

AST

Description:

Replace an Intent instantiation with null

How to find a mutant of this kind:

Provide a sequence of events that exercises the modified intent

Source Code level - Code Example APK level - Code Example

Before:

Intent intent = new Intent(main.this, ImportActivity.class);

After:

Intent intent = null;

Before:

new-instance v1, Landroid/content/Intent;
iget-object v2, p0, Lio/github/hidroh/materialistic/accounts/AccountAuthenticator;->mContext:Landroid/content/Context;
const-class v3, Lio/github/hidroh/materialistic/LoginActivity;
invoke-direct {v1, v2, v3}, Landroid/content/Intent;-><init>(Landroid/content/Context;Ljava/lang/Class;)V

After:

const/4 v1, 0x0

NullValueIntentPutExtra

AST

Description:

Replace the value argument in an Intent.putExtra(key, value) call with new Parcelable[0]

How to find a mutant of this kind:

Provide a sequence of events that exercises the modified intent

Source Code level - Code Example APK level - Code Example

Before:

intent.putExtra(key, value);

After:

intent.putExtra(key, new Parcelable[0]);

Before:

invoke-virtual {v0, v1, v9}, Landroid/content/Intent;->putExtra(Ljava/lang/String;Landroid/os/Parcelable;)Landroid/content/Intent;

After:

new-array v9, v9, [Landroid/os/Parcelable;
invoke-virtual {v0, v1, v9}, Landroid/content/Intent;->putExtra(Ljava/lang/String;Landroid/os/Parcelable;)Landroid/content/Intent;

WrongMainActivity

Text

Description:

Randomly replace the main activity definition with a different activity

How to find a mutant of this kind:

Check at app start for content expected to appear on launching

Source Code level - Code Example APK level - Code Example

Before:

<activity android:name=".Mileage" android:theme="@android:style/Theme.NoTitleBar">
    <intent-filter>
	    <action android:name="android.intent.action.MAIN" />
		<category android:name="android.intent.category.LAUNCHER" />
	</intent-filter>
</activity>

After:

<activity android:name=".AboutActivity" android:theme="@android:style/Theme.NoTitleBar">
    <intent-filter>
	    <action android:name="android.intent.action.MAIN" />
		<category android:name="android.intent.category.LAUNCHER" />
	</intent-filter>
</activity>

Before:

<activity android:name=".Mileage" android:theme="@android:style/Theme.NoTitleBar">
    <intent-filter>
	    <action android:name="android.intent.action.MAIN" />
		<category android:name="android.intent.category.LAUNCHER" />
	</intent-filter>
</activity>

After:

<activity android:name=".AboutActivity" android:theme="@android:style/Theme.NoTitleBar">
    <intent-filter>
	    <action android:name="android.intent.action.MAIN" />
		<category android:name="android.intent.category.LAUNCHER" />
	</intent-filter>
</activity>

Category 2 - Connectivity

BluetoothAdapterAlwaysEnabled

AST

Description:

Replace a BluetoothAdapter.isEnabled() call with "true"

How to find a mutant of this kind:

Call the method that has the bluetooth status validation while the adapter is turned off

Source Code level - Code Example APK level - Code Example

Before:

BluetoothAdapter btAdapter = BluetoothAdapter.getDefaultAdapter();
if(btAdapter.isEnabled()){
	transmitData();
}

After:

BluetoothAdapter btAdapter = BluetoothAdapter.getDefaultAdapter();
if(true){
	transmitData();
}

Before:

invoke-virtual/range {v19 .. v19}, 
    Landroid/bluetooth/BluetoothAdapter;->isEnabled()Z
move-result v3
if-eqz v3, :cond_13

After:

NullBluetoothAdapter

AST

Description:

Replace a BluetoothAdapter instance with null

How to find a mutant of this kind:

Exercise the method where the mutation was performed

Source Code level - Code Example APK level - Code Example

Before:

BluetoothAdapter btAdapter = BluetoothAdapter.getDefaultAdapter();

After:

BluetoothAdapter btAdapter = null

Before:

invoke-static {}, Landroid/bluetooth/BluetoothAdapter;
    ->getDefaultAdapter()Landroid/bluetooth/BluetoothAdapter;
move-result-object v10

After:

invoke-static {}, Landroid/bluetooth/BluetoothAdapter;
    ->getDefaultAdapter()Landroid/bluetooth/BluetoothAdapter;
const/16 v10, 0x0

Category 3 - GUI

BuggyGUIListener

AST

Description:

Assign null to a listener

How to find a mutant of this kind:

Provide a sequence of events that exercise the GUI component related to the modified "onClickListener"

Source Code level - Code Example

Before:

private View.OnClickListener listener = new View.OnClickListener() {
    @Override
    public void onClick(View view) {
      clicksCount += 1;
    }
}

After:

private View.OnClickListener listener = null;

FindViewByIdReturnsNull

AST

Description:

Assign a variable (returned by Activity.findViewById) to null

How to find a mutant of this kind:

Execute a sequence of test steps that covers the branch where the nulled variable is used

Source Code level - Code Example APK level - Code Example

Before:

ImageButton loadButton = (ImageButton) findViewById(R.id.load_data_button);

After:

ImageButton loadButton = null;

Before:

invoke-virtual {p0, p1}, 
    Lio/github/hidroh/materialistic/AboutActivity;
    ->findViewById(I)Landroid/view/View;
move-result-object v0
check-cast v0, Landroid/widget/TextView;

After:

const/4 v1, 0x0

InvalidColor

Text

Description:

Randomly change colors in layout files

How to find a mutant of this kind:

Perform visual regression over GUI for a sequence of events that visits the activity where the modified color is used

Source Code level - Code Example APK level - Code Example

Before:

<item name="red" type="color">#FFFF4444</item>

After:

<item name="red" type="color">#FF669900</item>

Before:

<item name="red" type="color">#FFFF4444</item>

After:

<item name="red" type="color">#FF669900</item>

InvalidIdFindView

AST

Description:

Replace the id argument in an Activitity.findViewById call

How to find a mutant of this kind:

Execute a sequence of test steps that covers the branch where the modified variable value is used

Source Code level - Code Example APK level - Code Example

Before:

TextView emailTextView = (TextView) findViewById(R.id.EmailTextView);

After:

TextView emailTextView = (TextView) findViewById(839);

Before:

const v5, 0x7f0d007d
invoke-virtual {v4, v5}, 
    Landroid/view/View;
    ->findViewById(I)Landroid/view/View;

After:

const/16 v5, 0x99abad00
invoke-virtual {v4, v5}, 
    Landroid/view/View;->
    findViewById(I)Landroid/view/View;

InvalidViewFocus

AST

Description:

Randomly focus a GUI component

How to find a mutant of this kind:

TODOs

Source Code level - Code Example APK level - Code Example

Before:

EditText etGLNum1 = (EditText)findViewById(R.id.etGLNum1);

After:

EditText etGLNum1 = (EditText)findViewById(R.id.etGLNum1);
etGLNum1.requestFocus();

Before:

invoke-virtual {p0, v1}, 
    Lcom/fsck/k9/view/ClientCertificateSpinner;
    ->findViewById(I)Landroid/view/View;
move-result-object v1
check-cast v1, Landroid/widget/Button;
iput-object v1, p0, 
    Lcom/fsck/k9/view/ClientCertificateSpinner;
    ->mSelection:Landroid/widget/Button;

After:

invoke-virtual {p0, v1}, 
    Lcom/fsck/k9/view/ClientCertificateSpinner;
    ->findViewById(I)Landroid/view/View;
move-result-object v1
check-cast v1, Landroid/widget/Button;
iput-object v1, p0, 
    Lcom/fsck/k9/view/ClientCertificateSpinner;
    ->mSelection:Landroid/widget/Button;
iget-object v1, p0, 
    Lcom/fsck/k9/view/ClientCertificateSpinner;
    ->mSelection:Landroid/widget/Button;
invoke-virtual {v1}, 
    Landroid/widget/Button;->requestFocus()Z

ViewComponentNotVisible

AST

Description:

Set visible attribute (from a View) to false

How to find a mutant of this kind:

Define a set of steps thar visits the activity where the component should appear, and then check existence of component

Source Code level - Code Example APK level - Code Example

Before:

TextView emailTextView = (TextView) findViewById(R.id.EmailTextView);

After:

 TextView emailTextView = (TextView) findViewById(R.id.EmailTextView);
emailTextView.setVisibility(android.view.View.INVISIBLE);

Before:

invoke-virtual {p0, v1}, 
    Lcom/fsck/k9/view/ClientCertificateSpinner;
    ->findViewById(I)Landroid/view/View;
move-result-object v1
check-cast v1, Landroid/widget/ImageButton;
iput-object v1, p0, 
    Lcom/fsck/k9/view/ClientCertificateSpinner;
    ->mDeleteButton:Landroid/widget/ImageButton;
.line 58
iget-object v1, p0, 
    Lcom/fsck/k9/view/ClientCertificateSpinner;
    ->mDeleteButton:Landroid/widget/ImageButton;

After:

invoke-virtual {p0, v1}, 
    Lcom/fsck/k9/view/ClientCertificateSpinner;
    ->findViewById(I)Landroid/view/View;
move-result-object v1
check-cast v1, Landroid/widget/ImageButton;
iput-object v1, p0, 
    Lcom/fsck/k9/view/ClientCertificateSpinner;
    ->mDeleteButton:Landroid/widget/ImageButton;
.line 58
iget-object v1, p0, 
    Lcom/fsck/k9/view/ClientCertificateSpinner;
    ->mDeleteButton:Landroid/widget/ImageButton;
const/4 v9, 0x4
invoke-virtual {v1, v9}, 
    Lcom/fsck/k9/view/ClientCertificateSpinner;
    ->setVisibility(I)V

Category 4 - Database

ClosingNullCursor

AST

Description:

Assign a cursor to null before it is closed

How to find a mutant of this kind:

Call the method where the modification was done, check for correct handling of error

Source Code level - Code Example APK level - Code Example

Before:

dbcursor.close();

After:

 dbcursor = null;
dbcursor.close();

Before:

invoke-interface {v3}, 
    Landroid/database/Cursor;->close()V

After:

const/4 v3, 0x0
invoke-interface {v3}, 
    Landroid/database/Cursor;->close()V

InvalidIndexQueryParameter

AST

Description:

Randomly modify indexes/order of query parameters

How to find a mutant of this kind:

Call the modified method and checks the correct handling of the error or empty result

Source Code level - Code Example APK level - Code Example

Before:

String[] args = new String[]{"user1", "user2"};
db.rawQuery("SELECT * FROM USER WHERE name=? OR name=?", args);

After:

String[] args = new String[]{"user1", "user2"};
db.rawQuery("SELECT * FROM USER WHERE name=? OR name=?", new String[]{ });

Before:

invoke-virtual {v5, v7, v4}, 
    Landroid/database/sqlite/SQLiteDatabase;->
    rawQuery(Ljava/lang/String;[Ljava/lang/String;)
    Landroid/database/Cursor;

After:

const/4 v1, 0x0
new-array v4, v1, [Ljava/lang/String;
invoke-virtual {v5, v7, v4}, 
    Landroid/database/sqlite/SQLiteDatabase;->
    rawQuery(Ljava/lang/String;[Ljava/lang/String;)
    Landroid/database/Cursor;

InvalidSQLQuery

AST

Description:

Randomly mutate a SQL query

How to find a mutant of this kind:

Call the modified method, checks for correct handling of error

Source Code level - Code Example APK level - Code Example

Before:

db.rawQuery("SELECT * FROM USER WHERE name=? OR name=?", args);

After:

db.rawQuery("ecab6839856b426fbdae3e6e8c46c38c", args);

Before:

invoke-virtual {v5, v7, v4}, 
    Landroid/database/sqlite/SQLiteDatabase;->
    rawQuery(Ljava/lang/String;[Ljava/lang/String;)
    Landroid/database/Cursor;

After:

const-string v7, "9e220601a5ff474caad7d5495b14af75"
invoke-virtual {v5, v7, v4}, 
    Landroid/database/sqlite/SQLiteDatabase;->
    rawQuery(Ljava/lang/String;[Ljava/lang/String;)
    Landroid/database/Cursor;

Category 5 - General Programming

InvalidDate

AST

Description:

Set a random Date to a date object

How to find a mutant of this kind:

Check integrity of data related to the modified date

Source Code level - Code Example APK level - Code Example

Before:

Date stdDate = Date(year, month, date);

After:

Date stdDate = Date(12345678910L);

Before:

invoke-direct {v6, v1, v2}, 
    Ljava/util/Date;-><init>(J)V

After:

const-wide v1, 0xeb34dd0L
invoke-direct {v6, v1, v2}, 
    Ljava/util/Date;-><init>(J)

InvalidMethodCallArgument

AST

Description:

Randomly mutate a method call argument of a basic type

How to find a mutant of this kind:

Call the modified method, check the correct handling of the error

Source Code level - Code Example

Before:

int transaction = getTransactionValue();
setValue(transaction);

After:

int transaction = getTransactionValue();
setValue(“string”);

NotSerializable

AST

Description:

Select a serializable class, remove "implements Serializable"

How to find a mutant of this kind:

Identify an element from the modified class, and try to exercise its serialization

Source Code level - Code Example

Before:

public class Book implements Serializable {
     private int mData;
 
	...
 
}

After:

public class Book {
     private int mData;
 
	...
 
}

NullMethodCallArgument

AST

Description:

Set null to a method call argument

How to find a mutant of this kind:

Call the modified method and check for correct handling of the error

Source Code level - Code Example APK level - Code Example

Before:

int transaction = getTransactionValue();
setValue(transaction);

After:

int transaction = getTransactionValue();
setValue(null);

Before:

invoke-direct {v0, v1, v5}, 
    Lcom/fsck/k9/Account$FolderMode;
    -><init>(Ljava/lang/String;I)V

After:

const/4 v1, 0x0
invoke-direct {v0, v1, v5}, 
    Lcom/fsck/k9/Account$FolderMode;
    -><init>(Ljava/lang/String;I)V

Category 6 - I/O

InvalidFilePath

AST

Description:

Randomly mutate paths to files

How to find a mutant of this kind:

Call for modified file usage and exercise it. Check for correct handling of the error

Source Code level - Code Example APK level - Code Example

Before:

File textFile = new File(“/sdcard/session.log”);

After:

File textFile = new File(“ecab6839856b426fbdae3e6e8c46c38c”);

Before:

invoke-direct {v0, v1}, 
    Ljava/io/File;-><init>(Ljava/lang/String;)V

After:

const-string v9, "e03044246cea40e195e920cd2eb9840f"
invoke-direct {v0, v9}, 
    Ljava/io/File;-><init>(Ljava/lang/String;)V

NullInputStream

AST

Description:

Assign an input stream to null before it is closed

How to find a mutant of this kind:

Call the modified method, checks for the correct handling of the error

Source Code level - Code Example APK level - Code Example

Before:

fileStream.close();

After:

fileStream = null;
fileStream.close();

Before:

invoke-virtual {v0}, Ljava/io/InputStream;->close()V

After:

const/4 v0, 0x0
invoke-virtual {v0}, Ljava/io/InputStream;->close()V

NullOutputStream

AST

Description:

Assign an output stream to null before it is closed

How to find a mutant of this kind:

Call the modified method, checks for the correct handling of the error

Source Code level - Code Example APK level - Code Example

Before:

outputStream.close();

After:

outputStream = null;
outputStream.close();

Before:

invoke-virtual {v0}, Ljava/io/OutputStream;->close()V

After:

const/4 v0, 0x0
invoke-virtual {v0}, Ljava/io/OutputStream;->close()V

Category 7 - Data

InvalidURI

AST

Description:

If URIs are used internally, randomly mutate the URIs

How to find a mutant of this kind:

Call for modified URI usage and exercise it. Check for correct handling of the error

Source Code level - Code Example APK level - Code Example

Before:

URI uri = new URI(u.toString());

After:

URI uri = new URI(“ecab6839856b426fbdae3e6e8c46c38c”);

Before:

invoke-direct {v3, v0}, 
    Ljava/net/URI;-><init>(Ljava/lang/String;)V

After:

const-string v2, "7e2cd5897951414db37193f8018a753c"
invoke-direct {v3, v2}, Ljava/net/URI;
    -><init>(Ljava/lang/String;)V

Category 8 - Non-Functional Requirements

LengthyBackEndService

AST

Description:

Inject large delay right-after a call to a back-end service. Current implementation add 10 second delay since a 5 second delay produces an ANR

How to find a mutant of this kind:

Check for correct handling of ARN

Source Code level - Code Example APK level - Code Example

Before:

HttpResponse response = client.execute(httpGet);
process(response);

After:

HttpResponse response = client.execute(httpGet);
try {
	Thread.sleep(10000);
} catch (InterruptedException e) {
	e.printStackTrace();
}
process(response);

Before:

.local v14, "response":Lorg/apache/http/HttpResponse;

After:

.local v14, "response":Lorg/apache/http/HttpResponse;
invoke-direct/range {p0 .. p0}, 
  Lcom/fsck/k9/mail/store/webdav/WebDavFolder;->delay()V
...
.method private delay()V
  .locals 2
  .line 332
  const-wide/16 v0, 0x2710
  :try_start_0
  invoke-static {v0, v1}, Ljava/lang/Thread;->sleep(J)V
  :try_end_0
  .catch Ljava/lang/InterruptedException; 
    {:try_start_0 .. :try_end_0} :catch_0
  .line 335
  goto :goto_0
  .line 333
  :catch_0
  move-exception v0
  .line 334
  .local v0, "e":Ljava/lang/InterruptedException;
  invoke-virtual {v0}, 
    Ljava/lang/InterruptedException;->printStackTrace()V
  .line 336
  .end local v0    # "e":Ljava/lang/InterruptedException;
  :goto_0
  return-void
.end method

LenghtyGUICreation

AST

Description:

Insert a long delay (\ie Thread.sleep(..)) in the creation GUI thread. Current implementation add 10 second delay since a 5 second delay produces an ANR

How to find a mutant of this kind:

Check for correct handling of ARN

Source Code level - Code Example APK level - Code Example

Before:

public void onCreate(Bundle savedInstanceState) {
    super.onCreate(savedInstanceState);
    setContentView(R.layout.main);
    Toast.makeText(getApplicationContext(),"2. onCreate()", Toast.LENGTH_SHORT).show();
}

After:

public void onCreate(Bundle savedInstanceState) {
    super.onCreate(savedInstanceState);    
    try {
        Thread.sleep(10000);
    } catch (InterruptedException e) {
        e.printStackTrace();
    }    
    setContentView(R.layout.main);
    Toast.makeText(getApplicationContext(),"2. onCreate()", Toast.LENGTH_SHORT).show();
}

Before:

.method public onCreate(Landroid/os/Bundle;)V
.locals 1
.param p1, "savedInstanceState"    # Landroid/os/Bundle;
.line 107
invoke-super {p0, p1}, Lcom/fsck/k9/activity/K9Activity;
    ->onCreate(Landroid/os/Bundle;)V

After:

.method public onCreate(Landroid/os/Bundle;)V
.locals 1
.param p1, "savedInstanceState"    # Landroid/os/Bundle;
.line 107
invoke-super {p0, p1}, Lcom/fsck/k9/activity/K9Activity;
  ->onCreate(Landroid/os/Bundle;)V
...
.method private delay()V
  .locals 2
  .line 332
  const-wide/16 v0, 0x2710
  :try_start_0
  invoke-static {v0, v1}, Ljava/lang/Thread;->sleep(J)V
  :try_end_0
  .catch Ljava/lang/InterruptedException; 
    {:try_start_0 .. :try_end_0} :catch_0
  .line 335
  goto :goto_0
  .line 333
  :catch_0
  move-exception v0
  .line 334
  .local v0, "e":Ljava/lang/InterruptedException;
  invoke-virtual {v0}, 
    Ljava/lang/InterruptedException;->printStackTrace()V
  .line 336
  .end local v0    # "e":Ljava/lang/InterruptedException;
  :goto_0
  return-void
.end method

LenghtyGUIListener

AST

Description:

Insert a long delay (\ie Thread.sleep(..)) in a GUI Listener

How to find a mutant of this kind:

Check for correct handling of ARN

Source Code level - Code Example APK level - Code Example

Before:

private View.OnClickListener listener = new View.OnClickListener() {
    @Override
    public void onClick(View view) {
        clicksCount += 1;
    }

After:

private View.OnClickListener listener = new View.OnClickListener() {
    @Override
    public void onClick(View view) {
        clicksCount += 1;
        try {
            Thread.sleep(10000);
        } catch (InterruptedException e) {
            e.printStackTrace();
        }
}

Before:

.method public onCreate(Landroid/os/Bundle;)V
.locals 1
.param p1, "v"    # Landroid/view/View;

After:

.method public onCreate(Landroid/os/Bundle;)V
.locals 1
.param p1, "v"    # Landroid/view/View;
...
.method private delay()V
  .locals 2
  .line 332
  const-wide/16 v0, 0x2710
  :try_start_0
  invoke-static {v0, v1}, Ljava/lang/Thread;->sleep(J)V
  :try_end_0
  .catch Ljava/lang/InterruptedException; 
    {:try_start_0 .. :try_end_0} :catch_0
  .line 335
  goto :goto_0
  .line 333
  :catch_0
  move-exception v0
  .line 334
  .local v0, "e":Ljava/lang/InterruptedException;
  invoke-virtual {v0}, 
    Ljava/lang/InterruptedException;->printStackTrace()V
  .line 336
  .end local v0    # "e":Ljava/lang/InterruptedException;
  :goto_0
  return-void
.end method

LongConnectionTimeOut

AST

Description:

Increase the time-out of connections to back-end services

How to find a mutant of this kind:

Check for correct handling of ARN

Source Code level - Code Example APK level - Code Example

Before:

HttpConnectionParams.setConnectionTimeout(my_httpParams, 3000);

After:

HttpConnectionParams.setConnectionTimeout(my_httpParams, 100000);

Before:

invoke-static {v3, v4}, 
    Lorg/apache/http/params/HttpConnectionParams;
    ->setConnectionTimeout(Lorg/apache/http/params/HttpParams;I)V

After:

const v4, 0x186a0
invoke-static {v3, v4}, 
    Lorg/apache/http/params/HttpConnectionParams;
    ->setConnectionTimeout(Lorg/apache/http/params/HttpParams;I)V

OOMLargeImage

AST

Description:

Increase the size of bitmaps by explicitly setting large dimensions. Current implementation modifies dimension into a size no device could fit

How to find a mutant of this kind:

Check for correct handling of oversized image

Source Code level - Code Example APK level - Code Example

Before:

tower = Bitmap.createScaledBitmap(BitmapFactory.decodeResource(res), width, height,true);

After:

tower = Bitmap.createScaledBitmap(BitmapFactory.decodeResource(res),100000,100000,true);

Before:

invoke-static {v8, v5, v6, v7}, 
    Landroid/graphics/Bitmap;
    ->createScaledBitmap(Landroid/graphics/Bitmap;IIZ)
    Landroid/graphics/Bitmap;

After:

const v6, 0x186a0
invoke-static {v8, v6, v6, v7}, 
    Landroid/graphics/Bitmap;
    ->createScaledBitmap(Landroid/graphics/Bitmap;IIZ)
    Landroid/graphics/Bitmap;

Category 9 - Android Programming

MissingPermissionManifest

Text

Description:

Select and remove an <uses-permission /> entry in the Manifest file

How to find a mutant of this kind:

Define a sequence of events that exercise the usage of the removed permission

Source Code level - Code Example APK level - Code Example

Before:

<uses-permission android:name="android.permission.WRITE_EXTERNAL_STORAGE" />

After:

Before:

<uses-permission android:name="android.permission.WRITE_EXTERNAL_STORAGE" />

After:

NotParcelable

AST

Description:

Select a parcelable class, remove "implements Parcelable" and the @override annotations

How to find a mutant of this kind:

Identify an element from the modified class, and try to use it inside a intend as parameter

Source Code level - Code Example

Before:

public class MyParcelable implements Parcelable {
    private int mData;
 
    @Override
    public int describeContents() {
        return 0;
    }
 
    @Override
    public void writeToParcel(Parcel out, int flags) {
        out.writeInt(mData);
    }
}

After:

public class MyParcelable {
    private int mData;
 
    public int describeContents() {
        return 0;
    }
    
    public void writeToParcel(Parcel out, int flags) {
        out.writeInt(mData);
    }
}

Before:

After:

NullGPSLocation

AST

Description:

Inject a Null GPS location in the location services

How to find a mutant of this kind:

Exercise the usage of the modified location variable

Source Code level - Code Example APK level - Code Example

Before:

Location GPSLocation = new Location(provider);

After:

Location GPSLocation = null;

Before:

.local v0, "location":Landroid/location/Location;

After:

const/4 v0, 0x0               
.local v0, "location":Landroid/location/Location;

SDKVersion

Text

Description:

Randomly mutate the integer values in the SdkVersion-related attributes

How to find a mutant of this kind:

Executes the app in a not-previously supported OS version

Source Code level - Code Example APK level - Code Example

Before:

<uses-sdk android:minSdkVersion="4" android:targetSdkVersion="10" />

After:

<uses-sdk android:minSdkVersion="21" android:targetSdkVersion="10" />

Before:

<uses-sdk android:minSdkVersion="4" android:targetSdkVersion="10" />

After:

<uses-sdk android:minSdkVersion="21" android:targetSdkVersion="10" />

WrongStringResource

Text

Description:

Select a <string /> entry in /res/values/strings.xml file and mutate the string value

How to find a mutant of this kind:

Identify a sequence of step that reach the activity where modified string is used and check for the expected value

Source Code level - Code Example APK level - Code Example

Before:

<string name="update_finished_importing">Import finished!</string>

After:

<string name="update_finished_importing">ecab6839856b426fbdae3e6e8c46c38c</string>

Before:

<string name="update_finished_importing">Import finished!</string>

After:

<string name="update_finished_importing">ecab6839856b426fbdae3e6e8c46c38c</string>

Category 10 - BackEnd Services

NullBackEndServiceReturn

AST

Description:

Assign null to a response variable from a back-end service

How to find a mutant of this kind:

Check for correct error handling

Source Code level - Code Example APK level - Code Example

Before:

HttpResponse response = client.execute(httpGet);

After:

HttpResponse response = null;

Before:

After:

Empirical Study

Apps used in Studies

App Name	Category	Package Name	Version	ESEC/FSE'17	TSE'20
A2DP Volume	Transportation	a2dp.Vol	2.8.11	✔	✔
AardDictionary	Books & Reference	aarddict.android	1.4.1	✔	✔
FTP Server	Tools	be.ppareit.swiftp_free	2.2	✔	✔
Bites	Lifestyle	caldwell.ben.bites	1.3	✔	✔
Battery Circle	Tools	ch.blinkenlights.battery	1.81	✔	✔
KeePassDroid	Tools	com.android.keepass	1.9.8	✔	✔
LolcatBuilder	Entertainment	com.android.lolcat	2	✔	✔
SpriteMethodTest	Sample	com.android.spritemethodtest	1	✔	✔
Alarm Clock	Tools	com.angrydoughnuts.android.alarmclock	1.7	✔	✔
Translate	Tools	com.beust.android.translate	1.6	✔	✔
Manpages	Productivity	com.chmod0.manpages	1.51	✔	✔
BookCatalogue	Productivity	com.eleybourn.bookcatalogue	3.8	✔	✔
Mileage	Finance	com.evancharlton.mileage	3.1.1	✔	✔
Auto Answer	Tools	com.everysoft.autoanswer	1.5	✔	✔
Amazed	Casual	com.example.amazed	2.0.2	✔	✔
RandomMusicPlayer	Music	com.example.android.musicplayer	1	✔	✔
AnyCut	Productivity	com.example.anycut	0.5	✔	✔
HNDroid	News & Magazines	com.gluegadget.hndroid	0.2.1	✔	✔
SpriteText	Sample	com.google.android.opengles.spritetext	-	✔	✔
Triangle	Sample	com.google.android.opengles.triangle	-	✔	✔
Photostream	Media & Video	com.google.android.photostream	1.1	✔	✔
Multi SMS	Communication	com.hectorone.multismssender	2.3	✔	✔
World Clock	Tools	com.irahul.worldclock	0.6	✔	✔
SyncMyPix	Media & Video	com.nloko.android.syncmypix	0.15	✔	✔
Jamendo	Music	com.teleca.jamendo	1.0.6-legacy	✔	✔
Yahtzee	Casual	com.tum.yahtzee	1	✔	✔
Sanity	Communication	cri.sanity	2.11	✔	✔
Mirrored	News & Magazines	de.homac.Mirrored	0.2.3	✔	✔
FileExplorer	Productivity	edu.killerud.fileexplorer	1	✔	✔
WeightChart	Health & Fitness	es.senselesssolutions.gpl.weightchart	1.0.4	✔	✔
SoundBoard	Sample	hiof.enigma.android.soundboard	1	✔	✔
ADSdroid	Books & Reference	hu.vsza.adsdroid	1.2	✔	✔
myLock	Tools	i4nc4mp.myLock	42	✔	✔
LockPatternGenerator	Tools	in.shick.lockpatterngenerator	2	✔	✔
MunchLife	Entertainment	info.bpace.munchlife	1.4.2	✔	✔
aGrep	Tools	jp.sblo.pandora.aGrep	0.2.1	✔	✔
CountdownTimer	Tools	net.everythingandroid.timer	1.1.0	✔	✔
LearnMusicNotes	Puzzle	net.fercanet.LNM	1.2	✔	✔
NetCounter	Tools	net.jaqpot.netcounter	0.14.1	✔	✔
TippyTipper	Finance	net.mandaria.tippytipper	1.1.3	✔	✔
BaterryDog	Tools	net.sf.andbatdog.batterydog	0.1.1	✔	✔
Bomber	Casual	org.beide.bomber	1	✔	✔
Dialer2	Productivity	org.dnaq.dialer2	2.9	✔	✔
FrozenBubble	Puzzle	org.jfedor.frozenbubble	1.12	✔	✔
aLogCat	Tools	org.jtb.alogcat	2.6.1	✔	✔
AnyMemo_135	Education	org.liberty.android.fantastischmemo	8.3.1	✔	✔
PasswordMakerPro	Tools	org.passwordmaker.android	1.1.7	✔	✔
Blokish	Puzzle	org.scoutant.blokish	2	✔	✔
ZooBorns	Entertainment	org.smerty.zooborns	1.4.4	✔	✔
Wordpress_394	Productivity	org.tomdroid	0.5.0	✔	✔
MyExpenses	Finance	org.totschnig.myexpenses	1.6.0	✔	✔
ImportContacts	Tools	org.waxworlds.edam.importcontacts	1.1	✔	✔
Wikipedia	Books & Reference	org.wikipedia	1.2.1	✔	✔

Operators Across Tools

Enabling Mutant Generation for Open- and Closed-Source Android Apps - TSE 2020

RQ1. Are the mutation operators (available for Java and Android apps) representative of real bugs in Android apps?

MDroid+ and MutAPK outperformed the other six mutation tools by achieving the highest coverage both in terms of bug types and bug instances. However, the results shows that Android- specific mutation operators should be combined with classic operators to generate mutants that are representative of real faults in mobile apps.

Bug Types not Covered by Taxonomy

Audio codec problem
Content provider (bulk update)
Improper implementation of Sensors as Activity
Improper location for invoking db insertions
Improper usage of static in Android
Issue with JNI
Issues with visibility (modifier)
Memory leaks
NegativeArraySizeException
Performance (large amount of data to insert in content provider)
Recycled bitmap
Text encoding error
Type error
Unreachable code
Usage of relative layout as the root container
Video encoding

RQ2. What is the rate of non-compilable (e.g., those leading to failed compilations), trivial (e.g., those leading to crashes on app launch), equivalent, and redundant mutants produced by the studied tools when used with Android apps?

First figure depicts the achieved results as percentage of Stillborn Mutants, the second figure presents the percentage of Trivial Mutants generated by each tool on each app. Third figure shows that on average, 67, 207, 1.3k+, 904, 2.6k+, and 1.5k+ mutants were generated by MDroid+, MutAPK-Shared, MutAPK, Major, PIT, and muDroid, respectively for each app. The larger number of mutants generated by PIT is due in part to the larger number of mutation operators available for the tool.

As for the generation of mutants, all the analyzed tools (Major, Pit, muDroid, MDroid+, MutAPK) generated a relatively low rate of trivial mutants, with muDroid being the worst with a 11.8% average rate of trivial mutants. Additionally, no equivalent mutants were found for any tool,according to a hash-based comparison between the original APKs and the corresponding mutants. Nevertheless, 4 tools (MutAPK, MutAPK-Shared, PIT and muDroid) generated redundant mutants, with muDroid being the worst with a 6.55% of total redundant mutants.

RQ3. What are the major causes for non-compilable, trivial, equivalent, and redundant mutants produced by the mutation testing tools when applied to Android apps?

The performed analysis indicate that the PIT tool outperforms others in terms of ratio between non-compilable and generated mutants, because it does not generate any non-compilable mutant. However,MDroid+ and MutAPK provide Android-specific mutations, which make the tools (i.e.,Pit, MDroid+, MutAPK) complementary for mutation testing of Android apps. MDroid+ and MutAPK generated the lowest rate of both non-compilable and trivial mutants (when compared to Major and muDroid), illustrating its immediate applicability to Android apps. Major and muDroid generate non-compilable mutants, with the latter having a critical average rate of 58.7% non-compilable mutants per app. Also, even when PIT generates redundant mutants, the number is insignificant when compared to the number of mutants generated; at the same time MutAPK and MutAPK-Shared also generate a low number of redundant mutants; some of them can be fixed by improving the current implementation.

RQ4. What are the benefits and trade-offs of performing mutation testing at APK level vs source code level?

The clear benefit of performing APK-level mutation analysis (MutAPK) as opposed to source code-level mutation analysis (MDroid+) relates primarily to the ease of use of the system, as it only requires a single file (i.e., an APKfile instead of several source files), and generates mutants with higher compilability ratio in less time. Moreover, this makes the mutation tool applicable to apps written with various languages, i.e., Java, Kotlin, and Dart. However, this ease of use comes with as light trade-off in terms of generating a higher number of mutants (which leads to an extensive execution effort from developers), and a higher number of trivial and redundant mutants for certain operators that are likely to be discarded during mutant analysis.

Enabling Mutation Testing for Android Apps - ESEC/FSE 2017

RQ1. Are the mutation operators (available for Java and Android apps) representative of real bugs in Android apps?

MDroid+ outperformed the other six mutation tools by achieving the highest coverage both in terms of bug types and bug instances. However, the results shows that Android- specific mutation operators should be combined with classic operators to generate mutants that are representative of real faults in mobile apps.

Bug Types not Covered by Taxonomy

Audio codec problem
Content provider (bulk update)
Improper implementation of Sensors as Activity
Improper location for invoking db insertions
Improper usage of static in Android
Issue with JNI
Issues with visibility (modifier)
Memory leaks
NegativeArraySizeException
Performance (large amount of data to insert in content provider)
Recycled bitmap
Text encoding error
Type error
Unreachable code
Usage of relative layout as the root container
Video encoding

RQ2. What is the rate of stillborn mutants (e.g., those leading to failed compilations) and trivial mutants (e.g., those leading to crashes on app launch) produced by the studied tools when used with Android apps?

First figure depicts the achieved results as percentage of Stillborn Mutants, the second figure presents the percentage of Trivial Mutants generated by each tool on each app. Third figure shows that on average, 167, 904, 2.6k+, and 1.5k+ mutants were generated by MDroid+, Major, PIT, and muDroid, respectively for each app. The larger number of mutants generated by PIT is due in part to the larger number of mutation operators available for the tool.

MDroid+ generated the smallest rate of both stillborn and trivial mutants illustrating its immediate applicability to Android apps. Major and muDroid generate stillborn mutants, with the latter having a critical average rate of 58.7% stillborn mutants per app.

RQ3. What is the rate of stillborn mutants (e.g., those leading to failed compilations) and trivial mutants (e.g., those leading to crashes on app launch) produced by the studied tools when used with Android apps?

All four tools generated a relatively low rate of trivial mutants, with muDroid again being the worst with an 11.8% average rate of trivial mutants. Our analysis shows that the PIT tool is most applicable to Android apps when evaluated in terms of the ratio between failed and generated mutants. However, MDroid+ is both practical and based on Android-specific operations implemented according to an empirically derived fault-taxonomy of Android apps.

ESEC/FSE 2017 - Study Data

Please click the button below to obtain a copy of the data generated for each Mutation framework relating to RQ2 & RQ3 outlined above.

Publications

MutAPK: Source-Codeless Mutant Generation for Android Apps (Tool Demo), Camilo Escobar-Velásquez, Michael Osorio-Riaño, and Mario Linares-Vásquez, The 34th IEEE/ACM International Conference on Automated Software Engineering (ASE'19), Demonstrations Track, San Diego, CA, USA, November 11th - 15th, 2019, to appear 4 pages (53.7% Acceptance Rate) [pdf][bibtex]
MDroid+: A Mutation Testing Framework for Android (Tool Demo), Kevin Moran, Michele Tufano, Carlos Bernal-Cárdenas, Mario Linares-Vásquez, Gabriele Bavota, Christopher Vendome, Massimiliano DiPenta, and Denys Poshyvanyk, The 40th IEEE/ACM International Conference on Software Engineering (ICSE'18), Formal Research Demonstrations Track, Gothenburg, Sweden, May 27-June 3, 2018, pp. 33-36 (35% Acceptance Rate) [pdf][bibtex]
Enabling Mutation Testing for Android Apps (Research Paper), Mario Linares-Vásquez, Gabriele Bavota, Michele Tufano, Kevin Moran, Massimiliano DiPenta, Christopher Vendome, Carlos Bernal-Cárdenas and Denys Poshyvanyk, The 11th Joint Meeting of the European Software Engineering Conference and the 25th ACM SIGSOFT Symposium on the Foundations of Software Engineering (ESEC/FSE'17), Paderborn, Germany, September 4-8, 2017, pp. 233-244 (24.4% Acceptance Rate) [pdf][bibtex]