Skip to content

Purchase IAP e2e test suite (#5186 follow-up)#5189

Merged
shai-almog merged 7 commits into
masterfrom
purchase-e2e-tests
Jun 6, 2026
Merged

Purchase IAP e2e test suite (#5186 follow-up)#5189
shai-almog merged 7 commits into
masterfrom
purchase-e2e-tests

Conversation

@shai-almog

@shai-almog shai-almog commented Jun 6, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Follow-up to the #5186 fix (#5188): a path-gated end-to-end test suite for the In-App-Purchase / ReceiptStore stack, verified locally on macOS and green in CI.

Layers (all green)

Layer What it does
Core receipt-sync (JavaSE) PurchaseTest regression guard: installs the store on one Purchase instance, drives the static Purchase.postReceipt(...) (which uses a different instance), asserts the receipt is submitted. Fails (expected 1 but was 0) without the #5186 static-receiptStore fix.
iOS StoreKitTest Hosted XCTest + SKTestSession drives a simulated App Store purchase through the real SKPaymentQueue observer → Purchase.postReceipt → receipt-sync → RecordingReceiptStore → NSUserDefaults sink, read back by the test.
Android billing-bridge Instrumentation test injects a fake IBillingSupport via a framework test seam, then drives purchase() → Purchase.postReceipt → RecordingReceiptStore, asserted via a CN1SS:IAP:SUBMITTED logcat marker.

Each platform exercises the same #5186 guard: a receipt submitted through a freshly-constructed Purchase instance still reaches the store installed on a different instance.

Dedicated test app (zero ripple)

The native layers build a dedicated minimal CN1 app (scripts/purchase-test-app/app) rather than the shared hellocodenameone sample — wiring IAP into the shared sample pulled Play Billing into every Android build and crashed the scripts-android screenshot suite. The dedicated app is built only by the native-ios / native-android purchase-e2e jobs, so StoreKit / Play Billing never ripple into other workflows. build-android-app.sh/build-ios-app.sh take an app dir via CN1_APP_DIR (defaults preserve hellocodenameone exactly).

Framework change

CodenameOneActivity.setBillingSupportTestOverride(...) — a tiny, test-only injection point (the generated stub shadows createBillingSupport() once the Purchase API is used). Not used in production.

CI

.github/workflows/purchase-e2e.yml, path-gated to the IAP surface: core-tests (container) + native-ios (macos) + native-android (emulator). Verified green, and Build Android/scripts-ios remain unaffected.

🤖 Generated with Claude Code

shai-almog and others added 2 commits June 6, 2026 15:49
@shai-almog @claude
Add #5186 regression test: ReceiptStore shared across fresh Purchase …
0376dce 
…instances

Layer 1 of the purchase e2e suite. The existing PurchaseTest pinned a single
cached Purchase via implementation.setInAppPurchase(), so it could not catch
the #5186 regression where a per-instance receiptStore is invisible to the
native postReceipt path (which arrives on a freshly-constructed instance, as
every real port returns a new Purchase per getInAppPurchase() call).

- TestCodenameOneImplementation: add an InAppPurchaseFactory hook so a test can
  reproduce the ports' fresh-instance-per-call behaviour.
- PurchaseTest.testReceiptStoreSharedAcrossFreshPurchaseInstances: install the
  store on one instance, drive Purchase.postReceipt(...) (which uses a different
  instance), and assert submitReceipt fires and the queue drains.

Verified: passes with the static-field fix; fails (expected 1 but was 0) when
the field is reverted to per-instance.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@shai-almog @claude
Add path-gated purchase-e2e workflow (core receipt-sync job)
6e35194 
Layer 5 (gating) + wiring Layer 1. Runs PurchaseTest (receipt sync state
machine + the #5186 fresh-instance regression guard) on the JavaSE simulator,
gated to the IAP surface: payment package, the iOS StoreKit observer +
ZoozPurchase/IOSImplementation, the Android BillingSupport/IBillingSupport/
ZoozPurchase/CodenameOneActivity, the payment unit tests, and the (incoming)
native test harness scripts. Modeled on identity-stack.yml.

native-ios (hosted XCTest + StoreKitTest) and native-android (fake
IBillingSupport instrumentation) jobs are added with their harnesses.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@github-actions

github-actions Bot commented Jun 6, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

✅ Continuous Quality Report

Test & Coverage

Static Analysis

  • SpotBugs [Report archive]
    • ✅ ByteCodeTranslator: 0 findings (no issues)
    • ✅ android: 0 findings (no issues)
    • ✅ codenameone-maven-plugin: 0 findings (no issues)
    • ✅ core-unittests: 0 findings (no issues)
    • ✅ ios: 0 findings (no issues)
  • ✅ PMD: 0 findings (no issues) [Report archive]
  • ✅ Checkstyle: 0 findings (no issues) [Report archive]

Generated automatically by the PR CI workflow.

shai-almog and others added 2 commits June 6, 2026 16:51
@shai-almog @claude
Add iOS StoreKitTest purchase e2e (Layers 2+3) to purchase-e2e suite
e006642 
Hosted XCTest driving a *simulated* App Store purchase via StoreKitTest's
SKTestSession on the simulator -- no sandbox account, no network. The purchase
flows through the real SKPaymentQueue into Codename One's actual StoreKit
observer (paymentQueue:updatedTransactions:), the generated Purchase.postReceipt,
the receipt-sync engine, and the ReceiptStore the sample installs at startup.
This is the iOS-level guard for #5186: the observer submits through a freshly
constructed Purchase instance, so a recorded submission proves the store
installed on a different instance is visible to it (shared/static receiptStore).

- Sample app (HelloCodenameOne.kt) references com.codename1.payment.Purchase so
  the builder flips usesPurchaseAPI -> defines CN1_USE_STOREKIT + links
  StoreKit.framework, and installs RecordingReceiptStore.
- RecordingReceiptStore forwards each submitted transactionId through the new
  PurchaseTestSink native interface; the iOS impl persists to NSUserDefaults
  (javase/android impls record in memory). The hosted XCTest reads it back.
- Products.storekit + PurchaseStoreKitTests.m drive SKTestSession.buyProduct and
  poll the sink. install-native-purchase-tests.sh injects the test, links
  StoreKit/StoreKitTest, bundles the config, configures the hosted target;
  run-ios-purchase-tests.sh orchestrates xcodebuild test.
- purchase-e2e.yml: add build-port + native-ios (macos-15) jobs.

Verified locally on macOS (Xcode 26.3, iOS 17 simulator): TEST SUCCEEDED, with
CN1SS:IAP:SUBMITTED emitted -- the StoreKitTest purchase reached the store.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@shai-almog @claude
Add Android billing-bridge purchase test (Layer 2) to purchase-e2e suite
6a6be6a 
There is no local Google Play Billing sandbox (no SKTestSession equivalent), so
this exercises the bridge with a fake IBillingSupport rather than a real
purchase.

- CodenameOneActivity: add a minimal test-only injection seam
  (setBillingSupportTestOverride) consulted by getBillingSupport(). Needed
  because the generated <Main>Stub auto-overrides createBillingSupport() +
  isBillingEnabled() whenever the app uses the Purchase API (our sample does,
  for the iOS wiring), which would otherwise shadow any subclass override. Not
  used in production.
- CN1TestBillingSupport (androidTest): fake whose initBilling()/purchase()
  synthesize a completed Play purchase via the static Purchase.postReceipt(...) --
  exactly where BillingSupport.onPurchasesUpdated would -- driving the receipt-
  sync engine and the app's RecordingReceiptStore (logs CN1SS:IAP:SUBMITTED).
- PurchaseBillingInstrumentationTest (androidTest): injects the fake, launches
  the app, and scrapes logcat for the marker. Android-side guard for #5186:
  postReceipt submits through a fresh Purchase instance, so the marker proves
  the store installed on a different instance is visible to it.
- purchase-e2e.yml: add native-android job (emulator) running only this test.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@shai-almog

shai-almog commented Jun 6, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator Author

Mac native screenshot updates

Compared 83 screenshots: 82 matched, 1 updated.

  • chart-transform — updated screenshot. Screenshot differs (1024x685 px, bit depth 8).

    chart-transform
    Preview info: JPEG preview quality 70; JPEG preview quality 70.
    Full-resolution PNG saved as chart-transform.png in workflow artifacts.

Benchmark Results

  • VM Translation Time: 0 seconds
  • Compilation Time: 134 seconds

@shai-almog

shai-almog commented Jun 6, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator Author

Compared 95 screenshots: 95 matched.
✅ JavaScript-port screenshot tests passed.

@shai-almog @claude
Android purchase test: license-key placeholder, drain-on-install, ver…
3fd536c 
…ified

Fixes to make the Android billing-bridge test pass on an emulator (verified
locally: connectedDebugAndroidTest BUILD SUCCESSFUL on cn1Api34Arm / API 34):

- codenameone_settings.properties: add a placeholder android.licenseKey. The
  sample referencing the Purchase API makes the Android builder require it
  (Play Billing); without it every Android build of the sample fails. The
  purchase test uses a fake IBillingSupport, so the key is never used to verify
  a real purchase -- it only needs to be defined.
- HelloCodenameOne.kt: after installing RecordingReceiptStore, call
  synchronizeReceipts() to drain receipts enqueued before the store existed.
  The fake fires a purchase from the activity's onCreate, which can race ahead
  of the CN1 init() that installs the store; draining on install makes it
  deterministic (and is sensible real-app behavior).
- CN1TestBillingSupport: log CN1SS:IAP_FAKE when firing, to aid CI diagnosis.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@shai-almog

shai-almog commented Jun 6, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator Author

Compared 124 screenshots: 124 matched.
✅ Native iOS Metal screenshot tests passed.

Benchmark Results

  • VM Translation Time: 0 seconds
  • Compilation Time: 179 seconds

Build and Run Timing

Metric Duration
Simulator Boot 55000 ms
Simulator Boot (Run) 0 ms
App Install 11000 ms
App Launch 3000 ms
Test Execution 233000 ms

Detailed Performance Metrics

Metric Duration
Base64 payload size 8192 bytes
Base64 benchmark iterations 6000
Base64 native encode 592.000 ms
Base64 CN1 encode 1279.000 ms
Base64 encode ratio (CN1/native) 2.160x (116.0% slower)
Base64 native decode 276.000 ms
Base64 CN1 decode 869.000 ms
Base64 decode ratio (CN1/native) 3.149x (214.9% slower)
Base64 SIMD encode 370.000 ms
Base64 encode ratio (SIMD/native) 0.625x (37.5% faster)
Base64 encode ratio (SIMD/CN1) 0.289x (71.1% faster)
Base64 SIMD decode 374.000 ms
Base64 decode ratio (SIMD/native) 1.355x (35.5% slower)
Base64 decode ratio (SIMD/CN1) 0.430x (57.0% faster)
Image encode benchmark iterations 100
Image createMask (SIMD off) 56.000 ms
Image createMask (SIMD on) 9.000 ms
Image createMask ratio (SIMD on/off) 0.161x (83.9% faster)
Image applyMask (SIMD off) 114.000 ms
Image applyMask (SIMD on) 50.000 ms
Image applyMask ratio (SIMD on/off) 0.439x (56.1% faster)
Image modifyAlpha (SIMD off) 150.000 ms
Image modifyAlpha (SIMD on) 73.000 ms
Image modifyAlpha ratio (SIMD on/off) 0.487x (51.3% faster)
Image modifyAlpha removeColor (SIMD off) 305.000 ms
Image modifyAlpha removeColor (SIMD on) 65.000 ms
Image modifyAlpha removeColor ratio (SIMD on/off) 0.213x (78.7% faster)
Image PNG encode (SIMD off) 1112.000 ms
Image PNG encode (SIMD on) 839.000 ms
Image PNG encode ratio (SIMD on/off) 0.754x (24.6% faster)
Image JPEG encode 461.000 ms

@shai-almog

shai-almog commented Jun 6, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator Author

Compared 124 screenshots: 124 matched.
✅ Native iOS screenshot tests passed.

Benchmark Results

  • VM Translation Time: 0 seconds
  • Compilation Time: 226 seconds

Build and Run Timing

Metric Duration
Simulator Boot 63000 ms
Simulator Boot (Run) 1000 ms
App Install 11000 ms
App Launch 11000 ms
Test Execution 316000 ms

Detailed Performance Metrics

Metric Duration
Base64 payload size 8192 bytes
Base64 benchmark iterations 6000
Base64 native encode 629.000 ms
Base64 CN1 encode 2220.000 ms
Base64 encode ratio (CN1/native) 3.529x (252.9% slower)
Base64 native decode 323.000 ms
Base64 CN1 decode 1387.000 ms
Base64 decode ratio (CN1/native) 4.294x (329.4% slower)
Base64 SIMD encode 682.000 ms
Base64 encode ratio (SIMD/native) 1.084x (8.4% slower)
Base64 encode ratio (SIMD/CN1) 0.307x (69.3% faster)
Base64 SIMD decode 546.000 ms
Base64 decode ratio (SIMD/native) 1.690x (69.0% slower)
Base64 decode ratio (SIMD/CN1) 0.394x (60.6% faster)
Image encode benchmark iterations 100
Image createMask (SIMD off) 58.000 ms
Image createMask (SIMD on) 9.000 ms
Image createMask ratio (SIMD on/off) 0.155x (84.5% faster)
Image applyMask (SIMD off) 124.000 ms
Image applyMask (SIMD on) 61.000 ms
Image applyMask ratio (SIMD on/off) 0.492x (50.8% faster)
Image modifyAlpha (SIMD off) 229.000 ms
Image modifyAlpha (SIMD on) 101.000 ms
Image modifyAlpha ratio (SIMD on/off) 0.441x (55.9% faster)
Image modifyAlpha removeColor (SIMD off) 251.000 ms
Image modifyAlpha removeColor (SIMD on) 63.000 ms
Image modifyAlpha removeColor ratio (SIMD on/off) 0.251x (74.9% faster)
Image PNG encode (SIMD off) 1010.000 ms
Image PNG encode (SIMD on) 946.000 ms
Image PNG encode ratio (SIMD on/off) 0.937x (6.3% faster)
Image JPEG encode 587.000 ms

@shai-almog @claude
Revert shared-sample IAP wiring; isolate Android purchase test (de-re…
16f71c3 
…gress CI)

Wiring IAP into the shared hellocodenameone sample broke the previously-green
scripts-android workflow: referencing the Purchase API pulled
com.android.billingclient into every Android build, and my purchase
instrumentation test (copied into every build from device-runner-app/androidTest)
ran inside the shared screenshot suite, crashing the instrumentation process.

Phase 1 - stop the bleeding:
- Revert HelloCodenameOne.kt + codenameone_settings.properties to master (no
  Purchase reference, no licenseKey) and remove the sample-side IAP files
  (PurchaseTestSink + impls, RecordingReceiptStore).
- Move the Android purchase test out of the auto-copied device-runner-app/
  androidTest into scripts/purchase-test-app/android-test-src so it no longer
  contaminates scripts-android.
- Scope purchase-e2e.yml to the verified core-tests job for now.

Kept: the core regression test (Layer 1, passing) and the harmless framework
test seam CodenameOneActivity.setBillingSupportTestOverride.

Phase 2 (next): a dedicated minimal IAP app built only by the purchase-e2e
iOS+Android jobs, so the StoreKitTest + billing-bridge tests run with zero
ripple to the shared sample's workflows.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@shai-almog

shai-almog commented Jun 6, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator Author

Compared 124 screenshots: 124 matched.

Native Android coverage

  • 📊 Line coverage: 13.23% (7915/59825 lines covered) [HTML preview] (artifact android-coverage-report, jacocoAndroidReport/html/index.html)
    • Other counters: instruction 10.71% (39184/365751), branch 4.58% (1577/34436), complexity 5.61% (1854/33035), method 9.83% (1520/15457), class 16.06% (348/2167)
    • Lowest covered classes
      • kotlin.collections.kotlin.collections.ArraysKt___ArraysKt – 0.00% (0/6327 lines covered)
      • kotlin.collections.unsigned.kotlin.collections.unsigned.UArraysKt___UArraysKt – 0.00% (0/2384 lines covered)
      • org.jacoco.agent.rt.internal_b6258fc.asm.org.jacoco.agent.rt.internal_b6258fc.asm.ClassReader – 0.00% (0/1519 lines covered)
      • kotlin.collections.kotlin.collections.CollectionsKt___CollectionsKt – 0.00% (0/1148 lines covered)
      • org.jacoco.agent.rt.internal_b6258fc.asm.org.jacoco.agent.rt.internal_b6258fc.asm.MethodWriter – 0.00% (0/923 lines covered)
      • kotlin.sequences.kotlin.sequences.SequencesKt___SequencesKt – 0.00% (0/730 lines covered)
      • kotlin.text.kotlin.text.StringsKt___StringsKt – 0.00% (0/623 lines covered)
      • org.jacoco.agent.rt.internal_b6258fc.asm.org.jacoco.agent.rt.internal_b6258fc.asm.Frame – 0.00% (0/564 lines covered)
      • kotlin.collections.kotlin.collections.ArraysKt___ArraysJvmKt – 0.00% (0/495 lines covered)
      • kotlinx.coroutines.kotlinx.coroutines.JobSupport – 0.00% (0/423 lines covered)

✅ Native Android screenshot tests passed.

Native Android coverage

  • 📊 Line coverage: 13.23% (7915/59825 lines covered) [HTML preview] (artifact android-coverage-report, jacocoAndroidReport/html/index.html)
    • Other counters: instruction 10.71% (39184/365751), branch 4.58% (1577/34436), complexity 5.61% (1854/33035), method 9.83% (1520/15457), class 16.06% (348/2167)
    • Lowest covered classes
      • kotlin.collections.kotlin.collections.ArraysKt___ArraysKt – 0.00% (0/6327 lines covered)
      • kotlin.collections.unsigned.kotlin.collections.unsigned.UArraysKt___UArraysKt – 0.00% (0/2384 lines covered)
      • org.jacoco.agent.rt.internal_b6258fc.asm.org.jacoco.agent.rt.internal_b6258fc.asm.ClassReader – 0.00% (0/1519 lines covered)
      • kotlin.collections.kotlin.collections.CollectionsKt___CollectionsKt – 0.00% (0/1148 lines covered)
      • org.jacoco.agent.rt.internal_b6258fc.asm.org.jacoco.agent.rt.internal_b6258fc.asm.MethodWriter – 0.00% (0/923 lines covered)
      • kotlin.sequences.kotlin.sequences.SequencesKt___SequencesKt – 0.00% (0/730 lines covered)
      • kotlin.text.kotlin.text.StringsKt___StringsKt – 0.00% (0/623 lines covered)
      • org.jacoco.agent.rt.internal_b6258fc.asm.org.jacoco.agent.rt.internal_b6258fc.asm.Frame – 0.00% (0/564 lines covered)
      • kotlin.collections.kotlin.collections.ArraysKt___ArraysJvmKt – 0.00% (0/495 lines covered)
      • kotlinx.coroutines.kotlinx.coroutines.JobSupport – 0.00% (0/423 lines covered)

Benchmark Results

Detailed Performance Metrics

Metric Duration
Base64 payload size 8192 bytes
Base64 benchmark iterations 6000
Base64 native encode 706.000 ms
Base64 CN1 encode 306.000 ms
Base64 encode ratio (CN1/native) 0.433x (56.7% faster)
Base64 native decode 889.000 ms
Base64 CN1 decode 275.000 ms
Base64 decode ratio (CN1/native) 0.309x (69.1% faster)
Image encode benchmark status skipped (SIMD unsupported)

@shai-almog @claude
Phase 2: dedicated IAP app for iOS+Android purchase tests (zero share…
a019c43 
…d-sample ripple)

Replaces the shared-hellocodenameone approach (which broke scripts-android) with
a dedicated minimal CN1 app at scripts/purchase-test-app/app that uses IAP and
installs RecordingReceiptStore. Built only by the purchase-e2e native jobs, so
StoreKit / Play Billing never ripple into the screenshot/notification workflows.

- scripts/purchase-test-app/app: minimal common/ios/android CN1 app (package
  com.codenameone.examples.purchasetest, mainName PurchaseTestApp). common wires
  IAP (references Purchase, installs RecordingReceiptStore, drains pending);
  PurchaseTestSink native interface (iOS NSUserDefaults impl + javase/android
  in-memory). Placeholder android.licenseKey (fake billing; never used at
  runtime). Trimmed to ios+android profiles only.
- scripts/purchase-test-app/android-test-src: CN1TestBillingSupport (fake
  IBillingSupport) + PurchaseBillingInstrumentationTest. The test injects the
  fake via the CodenameOneActivity seam, waits for the store-install marker,
  then drives purchase() explicitly (deterministic, no startup race) and scrapes
  logcat for CN1SS:IAP:SUBMITTED.
- build-android-app.sh: add CN1_APP_DIR + CN1_ANDROID_TEST_SOURCE_DIR support
  and derive the androidTest package from the app's codename1.packageName
  (defaults preserve hellocodenameone behaviour exactly). build-ios-app.sh
  already supported CN1_APP_DIR.
- purchase-e2e.yml: re-add native-ios (StoreKitTest) + native-android
  (billing-bridge) jobs, both building the dedicated app; fixed the
  android-emulator-runner cd+gradlew single-line invocation.

Verified locally on macOS: iOS xcodebuild test TEST SUCCEEDED; Android
connectedDebugAndroidTest BUILD SUCCESSFUL (cn1Api34Arm / API 34), both with
CN1SS:IAP:SUBMITTED. scripts-android/-ios unaffected (shared sample reverted).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@shai-almog shai-almog merged commit 1180bc2 into master Jun 6, 2026
22 checks passed
@shai-almog shai-almog deleted the purchase-e2e-tests branch June 6, 2026 17:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@shai-almog