Feature/storage/datalocality private drop - #50136
Draft
browndav-msft wants to merge 22 commits into
Draft
Conversation
* generate base files based on swagger docs * create live tests for createSession - downgrade blobserviceversion to 2026_04_06 - change AZURE_LIVE_TEST_SERVICE_VERSION to V2026_04_06 in ci.system.properties in azure-storage-common - create both sync and async * add recordings * create new files based on swagger update * add two params to BlobContainerClient#createSessionWithResponse * add sanitizers for SessionToken and SessionKey to BlobTestBase * add recording for createSessionReturnsTokenAndKey * create StorageSessionCredential with isExpired * create BlobSessionClient so that BlobSessionProvider takes it as a dep instead of ContainersImpl * create BlobSEssionClient with tests * add recorings for BlobSessionClient * fix BlobContainerAsyncClient to match new swagger, add new recording * add SessionProvider and SessionProviderTest * add accountName to BlobSessionClient * add accountName to StorageSessionCredential and SesionTestHelper * wip * change sessionprovider to SEssionTokenCredentialPolicy * wip * move session tests from containerapi to blobsessionclienttests * fix blobsessiontests and add place holder for end-to-end tests in containerapitests * add recordings for blobsessionclient * linting * refactor cache into separate class so it follows BearerTokenAuthenticationPolicy + AccessTokenCache pattern * add 503 fallback * add tests for udsas, but disabled for now * refactor createContext to use hardcoded endpoint * add SessionMode and tests for SessionMode * add sessionOptions to buildPipeline, add null to builders not using sessions * move SessionTokenCredentialPolicy ahead of StorageBearerTokenChallengeAuthorizationPolicy * fix linting issues * add session to BlobServiceClients and BlobServiceClientBuildeer * change expiration so that it defaults to 5 minutes, if there is no expiration * move SessionOptions so that it is public * remove old SessionOptions * remove unnecessary references to containerName and serviceVersion * add BlobContainerSessionInfo, add other Copilot recommendations * delete BlobContainerSessionInfo, restore return CreateSessionResponse * create createSession end-to-end test with recordings * only allow get requests for getblob * wrap tests in try-with-resources * make createSession package private * fixes based on copilot suggestions * Update sdk/storage/azure-storage-blob/src/main/java/com/azure/storage/blob/implementation/util/BlobSessionClient.java Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * add containerName to SessionOptions * move accountName to SessionOptions * refactor: SessionTokenCredentialPolicy accepts bearer policy as constructor dependency SessionTokenCredentialPolicy now takes StorageBearerTokenChallengeAuthorizationPolicy as a constructor dependency instead of relying on pipeline ordering to coordinate with the bearer policy. Key changes: - SessionTokenCredentialPolicy delegates to bearerPolicy.process() for non-session requests (non-GetBlob, NONE mode, AUTO first request) and fallback (503). - Added getBearerPolicy() accessor for use in per-container pipeline construction. - BuilderHelper updated to pass bearer policy through to SessionTokenCredentialPolicy constructor in both addSessionPolicyIfEnabled and wrapWithSessionPolicy. - Tests updated to mock bearerPolicy and verify delegation behavior. This is step 1 of the session auth refactor to align with the .NET SDK's SessionAuthenticationPolicy pattern. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * refactor: introduce AuthStrategy enum and consolidate analyzeRequest Consolidate isGetBlobRequest() and shouldUseSession() into a single analyzeRequest() method that returns an AuthStrategy enum (USE_BEARER_TOKEN or USE_SESSION_TOKEN), following the .NET SessionAuthenticationPolicy pattern. Also extract response handling into handleSessionResponse() and handleSessionResponseSync() methods for cleaner process()/processSync() flow. The process methods now have a clear structure: 1. analyzeRequest() -> decide strategy 2. USE_BEARER_TOKEN -> delegate to bearer policy 3. USE_SESSION_TOKEN -> acquire session, sign, send, handle response No behavioral changes - all existing tests pass unchanged. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * refactor: remove redundant restype check from analyzeRequest The restype query parameter check was redundant because requests with restype=container are container-level operations that have no blob name. They are already filtered by the blob name presence check. This aligns more closely with the .NET SessionAuthenticationPolicy which relies on URL structure rather than query parameter checks. The comp check is retained as a safety measure to exclude sub-operations like GetBlobMetadata (comp=metadata) that have a blob name but are not download operations. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * change sessionmode from always to singlespecifciedcontainer, add resolve method * wrap bearer token in sessioncredentialpolicy * fix NPE for SessionOptions, sessionoptions always non null * add tests for sessiontokencredpolicy and storagesessioncred * add logic to avoid wrapping Bearertoken, if session is not needed * add overloaded oauth in blobtestbase to be able to add sessionoptions * add overloaded getOAuthServiceAsyncClient to be able to pass session options * add custom buildStringToSign to remove `0` from get requests * readd versions * readd ci.system.properties * change session options check for null in BuilderHelper which affected other tests * add recordings for create sessions, change time to testResource time * add requestInspectionPolicy and overloaded getoauth client in base test * fix null sessionsoptions issue * add fix in storagesessioncredntial for query params * add SessionTokenCredPolicy to checks for anonymousAccess * remove constructor for BlobSessionClients that uses parse url * linting issues, remove SessionOptions from service methods * add comments to policyrefreshNearExpiry test * fix linting issues * add check for container name * fix return javadoc for SessionMode * add LOGGER and appropriate error throwing in BlobSessionClient * changes based on feedback from isabelle * add single retry for all 401 errors * remove unused imports * add suppression for SessionTokenPolicy for linting * fix ubuntu tests hanging by removing local dns bypass * add session for blob client with tests * create unified http transport between data requests and session request, add ResourceLock for live tests * test multiple concurrency * add branching if for bearer policy * refactor validateSessionMode to builderhelper * add revert to bearertoken for 400 errors * remove unused imports * readd comment to bearertokechallenge --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ta Locality Points swagger/README.md at the Data Locality fork spec (nickliu-msft/azure-rest-api-specs@5c678e4, 2026-10-06/blob.json) matching .NET PR Azure/azure-sdk-for-net#57554, and runs autorest to regenerate the implementation layer: new GetLayout REST operation (comp=layout), BlobLayout/BlobLayoutRanges/BlobLayoutEndpoints models, BlobsGetLayoutHeaders, and the new x-ms-download-hint header on BlobsDownloadHeaders. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: c4f5bab9-3ca0-444d-97cb-328fb210e7d7
Adds the paged getLayout / getLayoutAsync API for BlobClient/BlobAsyncClient (via BlobClientBase/BlobAsyncClientBase), returning BlobLayoutInfo with the blob's layout ranges/endpoints plus standard properties. New public types: BlobLayoutInfo, BlobLayoutRange, BlobGetLayoutOptions, DownloadHint. Wires the new x-ms-download-hint response header into BlobDownloadHeaders. ModelHelper.transformBlobLayoutInfo bridges the generated BlobLayout/BlobsGetLayoutHeaders response into the public BlobLayoutInfo model, reusing the existing getObjectReplicationDestinationPolicyId/getObjectReplicationSourcePolicies helpers for parity with other blob property transforms. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: c4f5bab9-3ca0-444d-97cb-328fb210e7d7
Sync (BlobClientBaseGetLayoutApiTests) and async (BlobClientBaseGetLayoutAsyncApiTests) coverage for BlobClientBase/BlobAsyncClientBase.getLayout: basic success, empty blob, ranged request, page-size limiting, continuation token, request-condition success/failure (mirroring PageBlobApiTests listPageRanges AC/ACFail pattern via BlobTestBase allConditionsSupplier/allConditionsFailSupplier), and a not-found error case. All tests gated with RequiredServiceVersion min 2027-03-07. No recordings yet -- fail as expected in playback with recording does not exist, which is normal at this stage. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: c4f5bab9-3ca0-444d-97cb-328fb210e7d7
Comprehensive explanation of what was built (getLayout paged API and DownloadHint signal, scoped to azure-storage-blob only), how it works (generated-layer wire protocol, hand-written model/client bridge, test coverage), and why key decisions were made (eager endpoint resolution in BlobLayoutRange, matching the existing listPageRanges pattern, deferring the data-locality routing infrastructure and DataLake module as follow-on work, using the unmerged fork spec to match the still-open .NET PR). Lists outstanding items: missing test recordings due to no credentials in this environment, the unmerged spec dependency, deferred routing/DataLake work, and a pending CHANGELOG entry. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: c4f5bab9-3ca0-444d-97cb-328fb210e7d7
The generated BlobsGetLayoutHeaders exposes both generic (Content-Length, Content-Type, etc from the layout response body) and blob-specific (x-ms-blob-content-length, x-ms-blob-content-type, etc reflecting the blobs actual properties) header pairs, matching the .NET model split between BlobLayoutInfo.ContentLength and BlobLayoutInfo.BlobContentLength/BlobContentType/BlobContentEncoding/BlobContentMD5/BlobCreatedOn. The initial Java BlobLayoutInfo only mapped the generic set, so getContentLength() returned null against a real service response. Adds the missing blobContentLength/blobContentType/blobContentEncoding/blobContentMd5/blobCreatedOn fields end to end (BlobLayoutInfo, ModelHelper.transformBlobLayoutInfo) and updates tests to assert on the correct getBlobContentLength(). Also fixes a StepVerifier logic bug in getLayoutAC (async): a successful conditional request legitimately emits one item, so the assertion needs response.then() before verifyComplete() to only assert on completion, not zero emissions -- matching the pattern already used by getLayoutContinuationToken/getLayoutRange in the same file. Found and fixed by actually recording against a real (preprod) storage account with AZURE_LIVE_TEST_SERVICE_VERSION=V2026_02_06. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: c4f5bab9-3ca0-444d-97cb-328fb210e7d7
…RVICE_VERSION afterTest built its cleanup BlobServiceClient directly from BlobServiceClientBuilder (httpClient/credential/endpoint only), bypassing the existing getServiceClient(TestAccount) helper that beforeTest already uses for every other client in this class. That helper routes through getServiceClientBuilder -> StorageCommonTestUtils.instrument, which is the only place that reads the AZURE_LIVE_TEST_SERVICE_VERSION environment variable and applies it via builder.serviceVersion(...). As a result, the cleanup listBlobContainers call always used the SDK's default/latest service version regardless of any environment override, causing every test to fail during teardown when running against an account that only supports an older pinned version (discovered while recording STG105 getLayout tests against a preprod account limited to V2026_02_06 -- the actual test bodies passed, only cleanup failed). Fix: reuse getServiceClient(ENVIRONMENT.getPrimaryAccount()) like the rest of the class. Pre-existing bug, unrelated to the STG105 feature itself, but blocking for anyone using AZURE_LIVE_TEST_SERVICE_VERSION in this module. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: c4f5bab9-3ca0-444d-97cb-328fb210e7d7
Java port of .NET's DataLocalityPolicy (Azure.Storage.Common/src/Shared/DataLocalityPolicy.cs): a no-op HttpPipelinePolicy unless a caller opts a request into locality-aware routing by setting DataLocalityPolicy.LAYOUT_ENDPOINT_KEY on the per-call Context to a non-empty endpoint URL. When set, rewrites the outgoing request's host/port to that endpoint (for routing to an alternate storage node returned by GetLayout) while preserving the original host as the Host header, so server-side virtual-hosting/TLS-SNI routing still works. Runs PER_RETRY (mirroring .NET's use on the raw HttpMessage before each send/retry). Malformed endpoint values are logged and skipped rather than failing the request. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: c4f5bab9-3ca0-444d-97cb-328fb210e7d7
Real HttpPipeline-based tests (no live service dependency) using a captured NoOpHttpClient and MockHttpResponse: no-op when context data is absent, rewrites host/port and preserves the original Host header when a layout endpoint is set, and no-op for an empty-string endpoint value. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: c4f5bab9-3ca0-444d-97cb-328fb210e7d7
…putStreamOptions Adds the EnableDataLocality-equivalent toggle (mirroring .NET's BlobDownloadToOptions/BlobOpenReadOptions) so callers can opt in to layout-aware routing for parallel file downloads and streamed reads. Wiring of the flag into the download/read paths is tracked separately. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: c4f5bab9-3ca0-444d-97cb-328fb210e7d7
…y routing Mirrors .NET's BlobLayoutSegmentCacheValue (cache wrapper distinguishing populated/empty/failed layout state) and the binary-search GetLayoutEndpoint helper from PR Azure/azure-sdk-for-net#57554, adapted to reuse the existing public BlobLayoutRange model (endpoint indices are already resolved by ModelHelper.transformBlobLayoutInfo) and the AutoRefreshingCache<T> added to azure-storage-common. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: c4f5bab9-3ca0-444d-97cb-328fb210e7d7
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: c4f5bab9-3ca0-444d-97cb-328fb210e7d7
Registers DataLocalityPolicy in the Blob HTTP pipeline (previously built but unregistered) and wires locality-aware routing into the two download entry points, opt-in via BlobDownloadToFileOptions / BlobInputStreamOptions.isEnableDataLocality(): - BlobAsyncClientBase.downloadToFileImpl: after the first chunk resolves, if enabled and the initial response's DownloadHint is LAYOUT, builds one AutoRefreshingCache<BlobLayoutCacheValue> per download (etag-locked, scoped to the remaining un-downloaded range) and resolves an endpoint per chunk via BlobLayoutRangeResolver, routing only chunks 1+ (chunk 0 already completed before the cache exists, matching the .NET PartitionedDownloader design). - BlobClientBase.openInputStream / BlobInputStream.dispatchRead: same pattern, scoped to the full requested stream range since reads can seek to arbitrary offsets. - Added a context-propagating internal getLayout(options, context) overload and fetchLayoutCacheValueAsync helper on BlobAsyncClientBase; soft-fails (caches a null/no-routing value) on BlobStorageException rather than failing the download. Mirrors Azure/azure-sdk-for-net#57554's PartitionedDownloader.cs and BlobBaseClient.DownloadStreamingInternal wiring. Default (enableDataLocality=false) behavior is unchanged; verified via BlobApiTests/BlobAsyncApiTests (738 tests) and BlobClientBaseGetLayout*ApiTests (38 tests), all passing. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: c4f5bab9-3ca0-444d-97cb-328fb210e7d7
Mirrors Azure/azure-sdk-for-net#57554's DataLakeFileClient.GetLayout and locality-aware routing, taking advantage of DataLakeFileClient already wrapping a BlockBlobClient internally: - New DataLakeFileLayoutInfo/DataLakeFileLayoutRange public models and DataLakeFileGetLayoutOptions options bag (DataLake-domain shapes, field-mapping conventions matched to Transforms.toPathProperties). - DataLakeFileClient/DataLakeFileAsyncClient.getLayout(...) proxies the already-implemented BlockBlobClient/BlockBlobAsyncClient getLayout API and maps BlobLayoutInfo -> DataLakeFileLayoutInfo via new Transforms helpers. Documented as an explicit proxy since DataLake does not yet have its own generated layout REST operation; revisit if/when one is added. - Added enableDataLocality to ReadToFileOptions/ DataLakeFileInputStreamOptions, threaded through to the underlying BlobDownloadToFileOptions/BlobInputStreamOptions. Since readToFileWithResponse/openInputStream already delegate to BlockBlobClient (which now has full locality routing wired in), this makes DataLake downloads/streams locality-aware with no additional chunking/routing logic needed on the DataLake side. Default (enableDataLocality=false) behavior is unchanged. Verified via FileApiTest/FileAsyncApiTests/FileInputStreamTests (827 tests), all passing; compile/checkstyle/spotbugs clean. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: c4f5bab9-3ca0-444d-97cb-328fb210e7d7
Adds BlobDataLocalityDownloadApiTests covering enableDataLocality on BlobDownloadToFileOptions/BlobInputStreamOptions: single-chunk and multi-chunk downloadToFileWithResponse, openInputStream (full and partial range), the enableDataLocality=false regression path, and that BlobProperties are still returned correctly. Recorded against a real (preprod) storage account with AZURE_LIVE_TEST_SERVICE_VERSION=V2026_02_06 (via the standard temporary TestEnvironment.java preprod patch, applied and reverted, not included in this commit). Finding: this account does not set x-ms-download-hint: Layout on plain download responses, so these recordings validate the wiring's regression-safety (the opt-in flag never changes downloaded bytes) end-to-end against a real service; the routing-selection logic itself (DownloadHint gating, per-chunk endpoint resolution, DataLocalityPolicy URL rewrite) remains covered by the existing unit tests for BlobLayoutRangeResolver, BlobLayoutCacheValue, AutoRefreshingCache, and DataLocalityPolicy. Verified passing in both RECORD and PLAYBACK (6/6). Recordings pushed to Azure/azure-sdk-assets as java/storage/azure-storage-blob_dfdc202e2d. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: c4f5bab9-3ca0-444d-97cb-328fb210e7d7
Documents the new AutoRefreshingCache/DataLocalityPolicy internal infrastructure (azure-storage-common), getLayout + enableDataLocality download/stream routing (azure-storage-blob), and the proxied getLayout + enableDataLocality pass-through (azure-storage-file-datalake), plus the 2027-03-07 service version support for the latter two. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: c4f5bab9-3ca0-444d-97cb-328fb210e7d7
|
Azure Pipelines: Successfully started running 1 pipeline(s). 33 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implement private drop for data locality