Automated Performance Hygiene: Integrating Baseline Profiles and Macrobenchmarks in CI/CD

Modern Android apps don't just need to work; they need to feel fast on first launch and stay smooth as users scroll and navigate. The hard part is making that performance repeatable and enforceable across releases and devices. In this post, we'll wire up an end-to-end, production-grade workflow that does exactly that: generate and ship Baseline Profiles to improve startup/jank, and run Macrobenchmarks in CI to catch regressions before they reach users.

You'll leave with:

• A working multi-module setup for Baseline Profiles and Macrobenchmarks
• Deterministic CI execution using Gradle Managed Devices
• Practical tests for cold startup and jank
• Real-world guardrails, pitfalls, and troubleshooting tips

Prerequisites and versions

• Android Studio Ladybug (2024.2.x) or newer
• JDK 17
• Gradle 8.6+
• AGP 8.5+
• Kotlin 2.0+
• Min SDK: 21+ (app); Test device API: 29+ (Macrobenchmark/MVD)
• Dependencies (use latest stable in your project; versions below are known stable as of late 2024):
  • androidx.profileinstaller:profileinstaller:1.3.1
  • androidx.benchmark:benchmark-macro-junit4:1.2.4
  • androidx.test.uiautomator:uiautomator:2.3.0
  • androidx.test.ext:junit:1.1.5
  • androidx.test:runner:1.5.2
  • Baseline Profile Gradle plugin: androidx.baselineprofile:baselineprofile-gradle-plugin:1.2.4

Note: Always prefer the latest stable artifacts from developer.android.com/jetpack/androidx/releases to benefit from fixes and device compatibility improvements.

Why this matters

• Baseline Profiles speed up your app by precompiling the hot code paths the first time the app is installed or updated. The result: significantly faster cold start and reduced jank right from v1 of a release.
• Macrobenchmarks measure real app performance (startup time, frame timing) on-device. They are your regression safety net.
• CI integration turns performance into a non-negotiable quality gate, not an afterthought.

We'll integrate both so every PR is validated against real-world performance - and we'll automatically produce and ship Baseline Profiles as part of your release pipeline.

What we'll build

We'll use a three-module layout:

• app - your production app (Compose or Views)
• baselineprofile - instrumentation tests that generate Baseline Profiles
• macrobenchmark - instrumentation tests that measure startup and jank

We'll also configure Gradle Managed Devices (MVD) to run both test suites in a reproducible emulator and wire CI to:

• Generate and update baseline-prof.txt for the app
• Run startup and frame-timing benchmarks
• Fail early if performance is broken

Step 1: Add ProfileInstaller to your app module

ProfileInstaller ships and installs your baseline-prof at app startup. Add it to app/build.gradle.kts:

plugins {
    id("com.android.application")
    kotlin("android")
}

android {
    namespace = "com.example.app"
    compileSdk = 35

    defaultConfig {
        applicationId = "com.example.app"
        minSdk = 21
        targetSdk = 35
        testInstrumentationRunner = "androidx.test.runner.AndroidJUnitRunner"
    }

    buildTypes {
        release {
            isMinifyEnabled = true
            proguardFiles(
                getDefaultProguardFile("proguard-android-optimize.txt"),
                "proguard-rules.pro"
            )
        }
        debug {
            // Keep debuggable true. Macrobench will never target debug anyway.
        }
    }
}

dependencies {
    implementation("androidx.profileinstaller:profileinstaller:1.3.1")
    // Usual app deps (Compose, etc.)
}

Tip:

• ProfileInstaller has consumer ProGuard rules; you usually don't need custom keep rules. If you use exotic shrinker configs, verify the content provider is not stripped.

Step 2: Create a Baseline Profile producer module

Add a new module "baselineprofile" of type "com.android.test". These instrumentation tests launch your app, execute critical user flows, and output a baseline profile.

baselineprofile/build.gradle.kts:

plugins {
    id("com.android.test")
    kotlin("android")
    // Baseline Profile Gradle Plugin - wires profile generation + copy
    id("androidx.baselineprofile") version "1.2.4"
}

android {
    namespace = "com.example.app.baselineprofile"
    compileSdk = 35

    defaultConfig {
        minSdk = 29 // required for Macrobenchmark/Baseline profile generation device
        testInstrumentationRunner = "androidx.test.runner.AndroidJUnitRunner"
        // Target the app under test
        targetProjectPath = ":app"
    }

    // Gradle Managed Devices for deterministic execution
    testOptions {
        managedDevices {
            devices {
                create<ManagedVirtualDevice>("pixel6Api31") {
                    device = "Pixel 6"
                    apiLevel = 31
                    systemImageSource = "google_apis" // Prefer google_apis for realistic perf
                }
            }
        }
        animationsDisabled = true
    }

    buildTypes {
        // Generate profiles against release variant because that's what you ship
        create("release")
    }
}

dependencies {
    implementation("androidx.benchmark:benchmark-macro-junit4:1.2.4")
    implementation("androidx.test.ext:junit:1.1.5")
    implementation("androidx.test:runner:1.5.2")
    implementation("androidx.test.uiautomator:uiautomator:2.3.0")
}

Baseline profile producer test (Kotlin):

package com.example.app.baselineprofile

import androidx.benchmark.macro.junit4.BaselineProfileRule
import androidx.test.ext.junit.runners.AndroidJUnit4
import org.junit.Rule
import org.junit.Test
import org.junit.runner.RunWith

private const val TARGET_PACKAGE = "com.example.app"

@RunWith(AndroidJUnit4::class)
class GenerateBaselineProfile {
    @get:Rule
    val rule = BaselineProfileRule()

    @Test
    fun generate() = rule.collect(
        packageName = TARGET_PACKAGE,
        // Run with release to collect a representative profile
        includeInStartupProfile = true
    ) {
        // 1) Cold start the default Activity
        startActivityAndWait()

        // 2) Execute your hot paths. Keep flows representative and deterministic.
        // Avoid sleeps; prefer waiting for idle or explicit conditions.

        // Example: navigate to Home -> Search -> Details and scroll
        device.waitForIdle()
        // Use UiAutomator or Compose test tags to interact reliably
        // device.findObject(By.res(TARGET_PACKAGE, "search")).click()
        // device.findObject(By.res(TARGET_PACKAGE, "query")).text = "kotlin"
        // device.pressEnter()
        // device.findObject(By.res(TARGET_PACKAGE, "result_0")).click()
        // device.swipe(x1, y1, x2, y2, steps)
    }
}

Notes:

• Keep this test focused on your most common warm paths (first-screen rendering + a couple of hot interactions).
• Prefer stable selectors (resource IDs or test tags) over text-based matches.
• Avoid randomness; performance tests must be deterministic.

Running it locally (managed device will be created and torn down automatically):

./gradlew :baselineprofile:pixel6Api31ReleaseAndroidTest

If you use the baseline profile plugin's task to orchestrate and copy into the app module, run:

./gradlew :baselineprofile:generateBaselineProfile

After generation, the plugin writes or updates baseline-prof.txt in your app:

• app/src/main/baseline-prof.txt (or per-flavor e.g. app/src/freeRelease/baseline-prof.txt)

Commit this file, just like ProGuard mappings or codegen outputs you curate.

Step 3: Add a Macrobenchmark module

This module contains startup and jank benchmarks. It targets your app's release variant and runs on API 29+ devices.

macrobenchmark/build.gradle.kts:

plugins {
    id("com.android.test")
    kotlin("android")
}

android {
    namespace = "com.example.app.macrobenchmark"
    compileSdk = 35

    defaultConfig {
        minSdk = 29
        testInstrumentationRunner = "androidx.test.runner.AndroidJUnitRunner"
        targetProjectPath = ":app"
    }

    testOptions {
        managedDevices {
            devices {
                create<ManagedVirtualDevice>("pixel6Api31") {
                    device = "Pixel 6"
                    apiLevel = 31
                    systemImageSource = "google_apis"
                }
            }
        }
        animationsDisabled = true
        execution = "ANDROIDX_TEST_ORCHESTRATOR"
    }

    // Ensure we measure release
    buildTypes {
        create("benchmark")
        // Map benchmark to release app if needed:
        // variantFilter { if (name != "benchmark") setIgnore(true) }
    }
}

dependencies {
    implementation("androidx.benchmark:benchmark-macro-junit4:1.2.4")
    implementation("androidx.test.ext:junit:1.1.5")
    implementation("androidx.test:runner:1.5.2")
    implementation("androidx.test.uiautomator:uiautomator:2.3.0")
    androidTestUtil("androidx.test:orchestrator:1.4.2")
}

Startup benchmark:

package com.example.app.macrobenchmark

import androidx.benchmark.macro.*
import androidx.test.ext.junit.runners.AndroidJUnit4
import org.junit.Rule
import org.junit.Test
import org.junit.runner.RunWith

private const val TARGET_PACKAGE = "com.example.app"

@RunWith(AndroidJUnit4::class)
class StartupBenchmarks {

    @get:Rule
    val benchmarkRule = MacrobenchmarkRule()

    // Cold startup with Baseline Profile required - fails if baseline is missing.
    @Test
    fun coldStartup_withBaselineProfile() = benchmarkRule.measureRepeated(
        packageName = TARGET_PACKAGE,
        metrics = listOf(StartupTimingMetric()),
        iterations = 10,
        startupMode = StartupMode.COLD,
        compilationMode = CompilationMode.Partial(
            baselineProfileMode = BaselineProfileMode.Require
        )
    ) {
        pressHome()
        startActivityAndWait()
    }

    // Compare a worst-case fallback (no pre-compilation)
    @Test
    fun coldStartup_noCompilation() = benchmarkRule.measureRepeated(
        packageName = TARGET_PACKAGE,
        metrics = listOf(StartupTimingMetric()),
        iterations = 5,
        startupMode = StartupMode.COLD,
        compilationMode = CompilationMode.None()
    ) {
        pressHome()
        startActivityAndWait()
    }
}

Jank/frame timing benchmark:

@RunWith(AndroidJUnit4::class)
class ScrollBenchmarks {

    @get:Rule
    val benchmarkRule = MacrobenchmarkRule()

    @Test
    fun homeFeed_scroll() = benchmarkRule.measureRepeated(
        packageName = TARGET_PACKAGE,
        metrics = listOf(FrameTimingMetric()),
        iterations = 5,
        compilationMode = CompilationMode.Partial(
            baselineProfileMode = BaselineProfileMode.UseIfAvailable
        ),
        setupBlock = {
            killProcess()
            startActivityAndWait()
        }
    ) {
        // Exercise a realistic scroll on your feed/list screen
        // Example using UiAutomator (replace with your IDs/test tags):
        // val recycler = device.findObject(By.res(TARGET_PACKAGE, "home_list"))
        // recycler.setGestureMargin(device.displayWidth / 10)
        // repeat(8) { recycler.fling(Direction.DOWN) }
        // repeat(2) { recycler.fling(Direction.UP) }
    }
}

Run locally:

./gradlew :macrobenchmark:pixel6Api31BenchmarkAndroidTest

You'll find Perfetto traces and JSON summaries under macrobenchmark/build/outputs/... for analysis.

Step 4: Deterministic devices with Gradle Managed Devices

Use MVD across both modules so CI can spin up the exact same emulator configuration every time. Key practices:

• Prefer "google_apis" images for more realistic performance counters.
• Disable animations (we used animationsDisabled = true).
• Pin one device model and API for consistency (e.g., Pixel 6 @ API 31).
• Keep tests short and deterministic to avoid thermal or scheduling drift.

Step 5: CI wiring (GitHub Actions example)

This workflow:

• Builds the app release
• Generates/updates Baseline Profiles
• Runs Macrobenchmarks
• Uploads benchmark artifacts

.github/workflows/perf.yml:

name: Performance hygiene

on:
  pull_request:
  push:
    branches: [ main ]

jobs:
  perf:
    runs-on: ubuntu-22.04

    steps:
      - uses: actions/checkout@v4

      - name: Setup JDK 17
        uses: actions/setup-java@v4
        with:
          distribution: temurin
          java-version: 17
          cache: gradle

      - name: Gradle info
        run: ./gradlew --version

      - name: Assemble release
        run: ./gradlew :app:assembleRelease

      # Generate and copy baseline-prof.txt into app/ (via plugin)
      - name: Generate Baseline Profile
        run: ./gradlew :baselineprofile:generateBaselineProfile --no-daemon --stacktrace

      - name: Validate app contains Baseline Profile
        run: |
          test -f app/src/main/baseline-prof.txt || (echo "Missing baseline-prof.txt"; exit 1)
          wc -l app/src/main/baseline-prof.txt

      # Run macrobenchmarks on the same managed device
      - name: Run Macrobenchmarks
        run: ./gradlew :macrobenchmark:pixel6Api31BenchmarkAndroidTest --no-daemon --stacktrace

      - name: Upload benchmark outputs
        uses: actions/upload-artifact@v4
        with:
          name: macrobenchmark-results
          path: macrobenchmark/build/outputs/**

Notes:

• The baseline profile plugin orchestrates tests and copies the profile file for you. If you prefer manual control, run the instrumentation test task and copy the generated baseline from the module's outputs to app/src/main/baseline-prof.txt.
• For gating on metric thresholds, see the next section.

Step 6: Gating on thresholds

Macrobenchmark currently reports results as files. A pragmatic approach is to parse its JSON summary in CI and fail the job when a metric exceeds your threshold. The file path can vary by AGP and device; look for summary JSON files under:

• macrobenchmark/build/outputs/managed_device_android_test/...
• macrobenchmark/build/outputs/androidTest-results/connected/...
• macrobenchmark/build/outputs/macrobenchmark/...

A simple gating step using jq might look like:

# Example path; adjust to your project's output
SUMMARY=$(ls macrobenchmark/build/outputs/**/macrobenchmark-*.json | head -n 1)

echo "Reading metrics from: $SUMMARY"

COLD_P50_MS=$(jq '.benchmarks[] | select(.name=="StartupBenchmarks_coldStartup_withBaselineProfile") | .metrics[] | select(.name=="startupMs") | .medianNs' "$SUMMARY" | awk '{printf "%.0f\n", $1/1000000}')

echo "cold-start p50: ${COLD_P50_MS}ms"
if [ "$COLD_P50_MS" -gt 900 ]; then
  echo "Regression: cold-start p50 > 900ms"
  exit 1
fi

Recommendations:

• Gate on medians or p90, not max.
• Reserve headroom for CI noise. Start generous (e.g., p50 < 1000ms) and tighten later.
• Record a weekly rolling baseline to monitor drift.

Step 7: Developer workflow

• Run profile generation locally before cutting a release:

./gradlew :baselineprofile:generateBaselineProfile
git add app/src/main/baseline-prof.txt

• Validate macrobench quickly:

./gradlew :macrobenchmark:pixel6Api31BenchmarkAndroidTest -Pandroid.experimental.testOptions.managedDevices.emulator=true

• If you introduce a new hot screen, update the profile producer test to exercise it.

Best practices and implementation notes

• Keep profile flows short and representative: launch, first frame, one or two critical navigations, and a simple scroll. Overly long or random flows produce noisy or brittle profiles.
• Target release builds for generation and measurement. Macrobenchmarks against debug variants are misleading.
• Use BaselineProfileMode.Require for at least one startup benchmark to ensure the shipped artifact contains a valid baseline. This will fail fast if the profile wasn't packaged.
• Compose apps: Prefer stable test tags and semantic actions. Avoid find-by-text for localization robustness.
• Variants/flavors: Baseline profiles can be flavor-specific. Place them under src/<flavor><BuildType>/baseline-prof.txt when necessary.
• Dynamic Feature Modules: Generate profiles that traverse into the feature. Include baseline profiles in the feature module as well if it ships separately.
• ART profile verification: After assembleRelease, inspect intermediates to ensure your baseline is packaged (e.g., app/build/intermediates/art-profile/release/ or use the plugin's verify tasks if available).
• Device stability:
  • Prefer one managed device configuration.
  • Ensure the runner disables animations.
  • Keep iteration counts modest to reduce thermal drift in CI.

Common issues and troubleshooting

• The Baseline Profile file didn't show up in the app module

  • Ensure the Baseline Profile Gradle Plugin is applied in the producer module and you executed its generate task.
  • Verify defaultConfig.targetProjectPath = ":app" in the producer module.
  • Check logs for "baseline-prof.txt" copy output; failing UI selectors may cause an empty profile.

• "BaselineProfileMode.Require" test fails

  • This means the app under test didn't include a baseline. Confirm app/src/main/baseline-prof.txt is present and that you're benchmarking the release variant. Rebuild the release and rerun tests.

• Macrobenchmark cannot find device or fails to boot

  • Use managed devices, not an arbitrary emulator started elsewhere.
  • Use a "google_apis" image. ATD images can lack some performance counters.
  • On CI, give the emulator time to boot; Gradle MVD handles this, but large images can still timeout if network is slow - bump Gradle's test timeouts if needed.

• Flaky UiAutomator selectors

  • Prefer resource IDs or Compose test tags. Wait for idle or specific view conditions instead of Thread.sleep.

• ProGuard/R8 stripping ProfileInstaller

  • Rare with modern versions. If in doubt, confirm the content provider exists via aapt dump badging on the release APK/AAB.

• "App startup is fast locally but slow in CI"

  • CI VMs are noisy. Gate on medians/p90 with safe headroom and keep device configuration stable.
  • Do not run resource-heavy jobs on the same runner concurrently.

What good looks like in production

• Baseline Profiles generated on every PR that touches hot paths; updated profiles are committed to main.
• Macrobenchmarks run on a stable managed device per PR and nightly on a second API level (e.g., API 31 and 34).
• CI gates prevent merges if:
  • BaselineProfiles are missing (Require mode fails),
  • Startup median exceeds your agreed threshold,
  • p90 frame time degrades past a tolerance band.
• Perfetto traces are uploaded as artifacts for debugging occasional regressions.

Key takeaways

• Baseline Profiles and Macrobenchmarks complement each other: one accelerates the app your users get, the other keeps you honest in CI.
• Treat performance like tests: deterministic devices, representative flows, and clear thresholds.
• Automate the boring parts. Let Gradle Managed Devices and the Baseline Profile plugin do the heavy lifting.
• Keep your baseline-prof.txt under version control and evolve it as your app's hot paths change.

Next steps:

• Add a second managed device (API 34, arm64) and compare results weekly.
• Split your macrobench suite into fast (PR) and deep (nightly) runs.
• Track trends by exporting metrics to your observability stack or a lightweight dashboard.

Ship fast - and stay fast.