-
Notifications
You must be signed in to change notification settings - Fork 1.5k
JAVA-5990 Hybrid Search Score Fusion #2024
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
nhachicha
wants to merge
27
commits into
mongodb:main
Choose a base branch
from
nhachicha:nh/hybrid_search
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
27 commits
Select commit
Hold shift + click to select a range
1bedd08
JAVA-5990 Add design spec for $scoreFusion aggregation stage builder
nhachicha 8f38f91
JAVA-5990 Add implementation plan for $scoreFusion builder
nhachicha beffb6d
JAVA-5990 Add ScoreNormalization for the $scoreFusion stage
nhachicha 6cf0342
JAVA-5990 Add FusionPipeline for fusion pipeline stages
nhachicha 41f428d
JAVA-5990 Add ScoreFusionCombination and ScoreFusionOptions
nhachicha e0182eb
JAVA-5990 Add Aggregates.scoreFusion pipeline stage builder
nhachicha 3f062bf
JAVA-5990 Add functional tests for the $scoreFusion stage
nhachicha a3049c1
JAVA-5990 Add Atlas hybrid search integration test for $scoreFusion
nhachicha a93e08e
JAVA-5990 Add Scala wrapper for the $scoreFusion stage builder
nhachicha 0498b71
JAVA-5990 Add Scala companion forwarders for scoreFusion types
nhachicha 5067a51
JAVA-5990 Reject blank fusion pipeline names and add validation tests
nhachicha c69cd62
JAVA-5990 Apply scalafmt formatting to scoreFusion forwarder
nhachicha 24376d8
Remove docs/superpowers from version control
nhachicha d735305
JAVA-5990 Align ScoreNormalization with JAVA-6202 enum implementation
nhachicha 3c4da1d
JAVA-5990 Assert fused scores in $scoreFusion functional tests
nhachicha e55d77d
JAVA-5990 Reference the $scoreFusion manual for the default avg combi…
nhachicha b88d01c
JAVA-5990 Correct sigmoid normalization range to (0, 1) in ScoreNorma…
nhachicha f0f5c42
JAVA-5990 Make ScoreNormalization docs stage-neutral
nhachicha 07beadd
JAVA-5990 Assert all scoreDetails fields in the $scoreFusion function…
nhachicha c5fca8c
JAVA-5990 Enforce server naming constraints for fusion pipeline names
nhachicha e12a387
JAVA-5990 Drop stale @Sealed from the ScoreNormalization Scala alias
nhachicha eba76c0
JAVA-5990 Align ScoreFusionCombination companion with ScoreFusionOpti…
nhachicha 1ab5d48
JAVA-5990 Strengthen assertions in the Atlas $scoreFusion integration…
nhachicha df3c366
JAVA-5990 Add Scala rendering tests for $scoreFusion and remove unuse…
nhachicha ffc5db4
Merge origin/main (JAVA-6202 $score stage) and drop the duplicate Sco…
nhachicha 91c7b3d
JAVA-5990 Rename idsFor to getIdsFor
nhachicha 8ea560b
JAVA-5990 Use the $score builder in $scoreFusion tests, resolving the…
nhachicha File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
123 changes: 123 additions & 0 deletions
123
driver-core/src/main/com/mongodb/client/model/FusionPipeline.java
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,123 @@ | ||
| /* | ||
| * Copyright 2008-present MongoDB, Inc. | ||
| * | ||
| * Licensed under the Apache License, Version 2.0 (the "License"); | ||
| * you may not use this file except in compliance with the License. | ||
| * You may obtain a copy of the License at | ||
| * | ||
| * http://www.apache.org/licenses/LICENSE-2.0 | ||
| * | ||
| * Unless required by applicable law or agreed to in writing, software | ||
| * distributed under the License is distributed on an "AS IS" BASIS, | ||
| * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| * See the License for the specific language governing permissions and | ||
| * limitations under the License. | ||
| */ | ||
|
|
||
| package com.mongodb.client.model; | ||
|
|
||
| import org.bson.conversions.Bson; | ||
|
|
||
| import java.util.ArrayList; | ||
| import java.util.List; | ||
| import java.util.Objects; | ||
|
|
||
| import static com.mongodb.assertions.Assertions.isTrueArgument; | ||
| import static com.mongodb.assertions.Assertions.notNull; | ||
| import static java.util.Arrays.asList; | ||
| import static java.util.Collections.unmodifiableList; | ||
|
|
||
| /** | ||
| * A named aggregation pipeline used as an input to a fusion pipeline stage, e.g., | ||
| * {@link Aggregates#scoreFusion(List, ScoreNormalization, ScoreFusionOptions) $scoreFusion}. | ||
| * The name uniquely identifies the pipeline within the stage and may be referred to | ||
| * by other parts of the stage, e.g., as the {@code "$$name"} variable in a | ||
| * {@linkplain ScoreFusionCombination#expression(Bson) combination expression}. | ||
| * A name must not be empty, must not start with {@code $}, and must not contain {@code .} | ||
| * or the null character. | ||
| * | ||
| * @since 5.10 | ||
| * @mongodb.server.release 8.2 | ||
| */ | ||
| public final class FusionPipeline { | ||
| private final String name; | ||
| private final List<Bson> pipeline; | ||
|
|
||
| /** | ||
| * Creates a new {@link FusionPipeline}. | ||
| * | ||
| * @param name The pipeline name, unique within the containing stage. It must not be empty, | ||
| * must not start with {@code $}, and must not contain {@code .} or the null character. | ||
| * @param pipeline The non-empty pipeline. | ||
| * @return The requested {@link FusionPipeline}. | ||
| */ | ||
| public static FusionPipeline of(final String name, final List<? extends Bson> pipeline) { | ||
| return new FusionPipeline(name, pipeline); | ||
| } | ||
|
|
||
| /** | ||
| * Creates a new {@link FusionPipeline}. | ||
| * | ||
| * @param name The pipeline name, unique within the containing stage. It must not be empty, | ||
| * must not start with {@code $}, and must not contain {@code .} or the null character. | ||
| * @param pipeline The non-empty pipeline. | ||
| * @return The requested {@link FusionPipeline}. | ||
| */ | ||
| public static FusionPipeline of(final String name, final Bson... pipeline) { | ||
| return new FusionPipeline(name, asList(pipeline)); | ||
| } | ||
|
|
||
| private FusionPipeline(final String name, final List<? extends Bson> pipeline) { | ||
| notNull("name", name); | ||
| isTrueArgument("name must not be empty", !name.trim().isEmpty()); | ||
| isTrueArgument("name must not start with '$'", !name.startsWith("$")); | ||
| isTrueArgument("name must not contain '.'", !name.contains(".")); | ||
| isTrueArgument("name must not contain the null character", name.indexOf('\u0000') < 0); | ||
| notNull("pipeline", pipeline); | ||
| isTrueArgument("pipeline must not be empty", !pipeline.isEmpty()); | ||
| for (Bson stage : pipeline) { | ||
| notNull("stage", stage); | ||
| } | ||
| this.name = name; | ||
| this.pipeline = unmodifiableList(new ArrayList<Bson>(pipeline)); | ||
| } | ||
|
|
||
| /** | ||
| * @return the pipeline name | ||
| */ | ||
| public String getName() { | ||
| return name; | ||
| } | ||
|
|
||
| /** | ||
| * @return the pipeline | ||
| */ | ||
| public List<? extends Bson> getPipeline() { | ||
| return pipeline; | ||
| } | ||
|
|
||
| @Override | ||
| public boolean equals(final Object o) { | ||
| if (this == o) { | ||
| return true; | ||
| } | ||
| if (o == null || getClass() != o.getClass()) { | ||
| return false; | ||
| } | ||
| FusionPipeline that = (FusionPipeline) o; | ||
| return name.equals(that.name) && pipeline.equals(that.pipeline); | ||
| } | ||
|
|
||
| @Override | ||
| public int hashCode() { | ||
| return Objects.hash(name, pipeline); | ||
| } | ||
|
|
||
| @Override | ||
| public String toString() { | ||
| return "FusionPipeline{" | ||
| + "name='" + name + '\'' | ||
| + ", pipeline=" + pipeline | ||
| + '}'; | ||
| } | ||
| } | ||
62 changes: 62 additions & 0 deletions
62
driver-core/src/main/com/mongodb/client/model/ScoreFusionCombination.java
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| @@ -0,0 +1,62 @@ | ||||||||||||
| /* | ||||||||||||
| * Copyright 2008-present MongoDB, Inc. | ||||||||||||
| * | ||||||||||||
| * Licensed under the Apache License, Version 2.0 (the "License"); | ||||||||||||
| * you may not use this file except in compliance with the License. | ||||||||||||
| * You may obtain a copy of the License at | ||||||||||||
| * | ||||||||||||
| * http://www.apache.org/licenses/LICENSE-2.0 | ||||||||||||
| * | ||||||||||||
| * Unless required by applicable law or agreed to in writing, software | ||||||||||||
| * distributed under the License is distributed on an "AS IS" BASIS, | ||||||||||||
| * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||||||||||||
| * See the License for the specific language governing permissions and | ||||||||||||
| * limitations under the License. | ||||||||||||
| */ | ||||||||||||
|
|
||||||||||||
| package com.mongodb.client.model; | ||||||||||||
|
|
||||||||||||
| import com.mongodb.annotations.Sealed; | ||||||||||||
| import org.bson.BsonString; | ||||||||||||
| import org.bson.Document; | ||||||||||||
| import org.bson.conversions.Bson; | ||||||||||||
|
|
||||||||||||
| import static com.mongodb.assertions.Assertions.notNull; | ||||||||||||
|
|
||||||||||||
| /** | ||||||||||||
| * The way in which the normalized scores produced by the input pipelines of the | ||||||||||||
| * {@linkplain Aggregates#scoreFusion(java.util.List, ScoreNormalization, ScoreFusionOptions) $scoreFusion} | ||||||||||||
| * stage are combined into the final score. The server rejects specifying both | ||||||||||||
| * {@linkplain #weighted(Bson) weights} and an {@linkplain #expression(Bson) expression}, | ||||||||||||
| * which is why they are separate factory methods. | ||||||||||||
| * | ||||||||||||
| * @see ScoreFusionOptions#combination(ScoreFusionCombination) | ||||||||||||
| * @since 5.10 | ||||||||||||
| * @mongodb.server.release 8.2 | ||||||||||||
|
Comment on lines
+34
to
+35
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Add manual link
Suggested change
|
||||||||||||
| */ | ||||||||||||
| @Sealed | ||||||||||||
| public interface ScoreFusionCombination extends Bson { | ||||||||||||
| /** | ||||||||||||
| * Returns a {@link WeightedScoreFusionCombination} combining the scores using per-pipeline weights. | ||||||||||||
| * | ||||||||||||
| * @param weights A document mapping {@linkplain FusionPipeline#getName() pipeline names} to non-negative | ||||||||||||
| * numeric weights. Pipelines not mentioned have the server-default weight 1. | ||||||||||||
| * @return The requested {@link WeightedScoreFusionCombination}. | ||||||||||||
| */ | ||||||||||||
| static WeightedScoreFusionCombination weighted(final Bson weights) { | ||||||||||||
| return new ScoreFusionConstructibleBson(new Document("weights", notNull("weights", weights))); | ||||||||||||
| } | ||||||||||||
|
|
||||||||||||
| /** | ||||||||||||
| * Returns a {@link ScoreFusionCombination} combining the scores using a custom expression. | ||||||||||||
| * The normalized, weighted score of each input pipeline is available to the expression | ||||||||||||
| * as the variable named after the pipeline, e.g., {@code "$$name"}. | ||||||||||||
| * | ||||||||||||
| * @param expression The combination expression. | ||||||||||||
| * @return The requested {@link ScoreFusionCombination}. | ||||||||||||
| */ | ||||||||||||
| static ScoreFusionCombination expression(final Bson expression) { | ||||||||||||
| return new ScoreFusionConstructibleBson(new Document("method", new BsonString("expression")) | ||||||||||||
| .append("expression", notNull("expression", expression))); | ||||||||||||
| } | ||||||||||||
| } | ||||||||||||
69 changes: 69 additions & 0 deletions
69
driver-core/src/main/com/mongodb/client/model/ScoreFusionConstructibleBson.java
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,69 @@ | ||
| /* | ||
| * Copyright 2008-present MongoDB, Inc. | ||
| * | ||
| * Licensed under the Apache License, Version 2.0 (the "License"); | ||
| * you may not use this file except in compliance with the License. | ||
| * You may obtain a copy of the License at | ||
| * | ||
| * http://www.apache.org/licenses/LICENSE-2.0 | ||
| * | ||
| * Unless required by applicable law or agreed to in writing, software | ||
| * distributed under the License is distributed on an "AS IS" BASIS, | ||
| * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| * See the License for the specific language governing permissions and | ||
| * limitations under the License. | ||
| */ | ||
|
|
||
| package com.mongodb.client.model; | ||
|
|
||
| import com.mongodb.annotations.Immutable; | ||
| import com.mongodb.internal.client.model.AbstractConstructibleBson; | ||
| import org.bson.BsonBoolean; | ||
| import org.bson.BsonDocument; | ||
| import org.bson.BsonString; | ||
| import org.bson.Document; | ||
| import org.bson.conversions.Bson; | ||
|
|
||
| import static com.mongodb.assertions.Assertions.notNull; | ||
|
|
||
| final class ScoreFusionConstructibleBson extends AbstractConstructibleBson<ScoreFusionConstructibleBson> | ||
| implements ScoreFusionOptions, WeightedScoreFusionCombination { | ||
| /** | ||
| * An {@linkplain Immutable immutable} {@link BsonDocument#isEmpty() empty} instance. | ||
| */ | ||
| static final ScoreFusionConstructibleBson EMPTY_IMMUTABLE = | ||
| new ScoreFusionConstructibleBson(AbstractConstructibleBson.EMPTY_IMMUTABLE); | ||
|
|
||
| ScoreFusionConstructibleBson(final Bson base) { | ||
| super(base); | ||
| } | ||
|
|
||
| private ScoreFusionConstructibleBson(final Bson base, final Document appended) { | ||
| super(base, appended); | ||
| } | ||
|
|
||
| @Override | ||
| protected ScoreFusionConstructibleBson newSelf(final Bson base, final Document appended) { | ||
| return new ScoreFusionConstructibleBson(base, appended); | ||
| } | ||
|
|
||
| @Override | ||
| public ScoreFusionOptions combination(final ScoreFusionCombination combination) { | ||
| return newAppended("combination", notNull("combination", combination)); | ||
| } | ||
|
|
||
| @Override | ||
| public ScoreFusionOptions scoreDetails(final boolean scoreDetails) { | ||
| return newAppended("scoreDetails", BsonBoolean.valueOf(scoreDetails)); | ||
| } | ||
|
|
||
| @Override | ||
| public ScoreFusionOptions option(final String name, final Object value) { | ||
| return newAppended(notNull("name", name), notNull("value", value)); | ||
| } | ||
|
|
||
| @Override | ||
| public WeightedScoreFusionCombination avg() { | ||
| return newAppended("method", new BsonString("avg")); | ||
| } | ||
| } |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nit: