feat(huggingface): add qa and ranking tasks

[anishshiva7]

What changes were proposed in this PR?

Adds the QA/ranking/classification task family — 5 HF pipeline tasks — as a new TaskCodegen plugged into the dispatcher established by the text-generation PR:

QA tasks: question-answering, table-question-answering

classification/ranking tasks: zero-shot-classification, sentence-similarity, text-ranking

codegen/QaRankingCodegen.scala supplies the per-task payload + parse Python branches for all 5 tasks.

CodegenContext is extended with contextColumn, candidateLabels, and sentencesColumn (EncodableString).

HuggingFaceInferenceOpDesc.scala gains 3 new @JsonProperty fields and registers QaRankingCodegen in the dispatcher.

PythonCodegenBase.scala grows to host the shared QA/ranking infrastructure:

• Per-row validation for the new column-named fields.
• question-answering payload handling with prompt + context.
• table-question-answering payload handling with table data.
• zero-shot-classification payload handling with candidate labels.
• sentence-similarity and text-ranking payload handling with sentence inputs.
• Response parsing for QA/ranking outputs.

User-input strings continue to flow through pyb"..." + EncodableString so they reach Python as self.decode_python_template('<base64>') rather than raw literals. PythonCodeRawInvalidTextSpec still passes with 117/117 descriptors py_compile cleanly.

Any related issues, documentation, or discussions?

Tracking issue: Add HuggingFace question answering and ranking tasks #5292

Closes #5292

Stacked on: PR 4 audio/media generation tasks / hf/04-audio-mediagen

Parent issue: Add Hugging Face inference operator #5041

Closed sibling issue: Add HuggingFaceModelResource REST endpoints for HF operator UI #5134

How was this PR tested?

sbt "WorkflowOperator/compile; WorkflowOperator/Test/compile" clean.

sbt "WorkflowOperator/testOnly org.apache.texera.amber.operator.huggingFace.HuggingFaceInferenceOpDescSpec org.apache.texera.amber.util.PythonCodeRawInvalidTextSpec" — 31 focused tests pass, including HuggingFace QA/ranking task coverage and the raw Python descriptor scan.

sbt "WorkflowOperator / scalafmtCheck" clean.

sbt "WorkflowOperator / Test / scalafmtCheck" clean.

PythonCodeRawInvalidTextSpec — 117/117 descriptors py_compile cleanly with the new operator code paths, no marker leaks.

Was this PR authored or co-authored using generative AI tooling?

Yes, co-authored with generative AI tooling (Codex).

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 54.10%. Comparing base (439ea72) to head (cb2bd44).

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #5574      +/-   ##
============================================
+ Coverage     54.09%   54.10%   +0.01%     
- Complexity     2817     2818       +1     
============================================
  Files          1103     1104       +1     
  Lines         42650    42679      +29     
  Branches       4588     4591       +3     
============================================
+ Hits          23070    23090      +20     
- Misses        18236    18244       +8     
- Partials       1344     1345       +1     

Flag	Coverage Δ		*Carryforward flag
access-control-service	70.44% <ø> (ø)
agent-service	34.36% <ø> (ø)		Carriedforward from 439ea72
amber	55.64% <100.00%> (+0.02%)	⬆️
computing-unit-managing-service	1.65% <ø> (ø)
config-service	57.35% <ø> (ø)
file-service	58.59% <ø> (ø)
frontend	48.09% <ø> (ø)		Carriedforward from 439ea72
pyamber	90.20% <ø> (ø)		Carriedforward from 439ea72
python	90.76% <ø> (ø)		Carriedforward from 439ea72
workflow-compiling-service	58.69% <ø> (ø)

*This pull request uses carry forward flags. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[anishshiva7]

/request-review @Ma77Ball

[Ma77Ball]

Left comments below

[github-actions]

⚠️ Benchmark changes need a look

🟢 2 better · 🔴 5 worse · ⚪ 8 noise (<±5%) · 0 without baseline

Compared against main 439ea72 benchmarked on this same runner, so the delta is largely free of cross-runner hardware noise. The "7d avg" column still reflects the gh-pages dashboard. Treat <±5% as noise unless repeated.

Dashboard · Run

	config	throughput	MB/s	latency	max Δ latest / 7d
🔴	bs=10 sw=10 sl=64	377	0.23	25,621/32,668/32,668 us	🔴 +20.7% / 🔴 -8.3%
🔴	bs=100 sw=10 sl=64	804	0.491	120,319/172,171/172,171 us	🔴 +6.8% / 🔴 +23.2%
⚪	bs=1000 sw=10 sl=64	932	0.569	1,072,268/1,137,669/1,137,669 us	⚪ within ±5% / 🔴 +11.2%

Baseline details

Latest main 439ea72 from same runner

config	metric	PR	latest main	7d avg	Δ latest	Δ 7d
bs=10 sw=10 sl=64	throughput	377 tuples/sec	442 tuples/sec	410.82 tuples/sec	-14.7%	-8.2%
bs=10 sw=10 sl=64	MB/s	0.23 MB/s	0.27 MB/s	0.251 MB/s	-14.8%	-8.3%
bs=10 sw=10 sl=64	p50	25,621 us	21,222 us	23,785 us	+20.7%	+7.7%
bs=10 sw=10 sl=64	p95	32,668 us	34,399 us	34,980 us	-5.0%	-6.6%
bs=10 sw=10 sl=64	p99	32,668 us	34,399 us	34,980 us	-5.0%	-6.6%
bs=100 sw=10 sl=64	throughput	804 tuples/sec	809 tuples/sec	891.94 tuples/sec	-0.6%	-9.9%
bs=100 sw=10 sl=64	MB/s	0.491 MB/s	0.494 MB/s	0.544 MB/s	-0.6%	-9.8%
bs=100 sw=10 sl=64	p50	120,319 us	119,336 us	112,277 us	+0.8%	+7.2%
bs=100 sw=10 sl=64	p95	172,171 us	161,155 us	139,802 us	+6.8%	+23.2%
bs=100 sw=10 sl=64	p99	172,171 us	161,155 us	139,802 us	+6.8%	+23.2%
bs=1000 sw=10 sl=64	throughput	932 tuples/sec	935 tuples/sec	1,041 tuples/sec	-0.3%	-10.5%
bs=1000 sw=10 sl=64	MB/s	0.569 MB/s	0.571 MB/s	0.635 MB/s	-0.4%	-10.4%
bs=1000 sw=10 sl=64	p50	1,072,268 us	1,073,831 us	972,714 us	-0.1%	+10.2%
bs=1000 sw=10 sl=64	p95	1,137,669 us	1,103,851 us	1,023,057 us	+3.1%	+11.2%
bs=1000 sw=10 sl=64	p99	1,137,669 us	1,103,851 us	1,023,057 us	+3.1%	+11.2%

Raw CSV

config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,530.56,200,128000,377,0.230,25620.89,32667.95,32667.95
1,100,10,64,20,2488.02,2000,1280000,804,0.491,120319.12,172171.24,172171.24
2,1000,10,64,20,21465.84,20000,12800000,932,0.569,1072268.20,1137668.58,1137668.58

[anishshiva7]

/request-review @Ma77Ball

[Copilot]

Pull request overview

Adds additional HuggingFace task-family support to the workflow-operator HuggingFace inference operator by extending the Scala descriptor/codegen dispatcher and the shared Python codegen template, plus a small JWT parsing hardening improvement in auth.

Changes:

• Add QA/ranking/classification task family support via new QaRankingCodegen and new descriptor/context fields (contextColumn, candidateLabels, sentencesColumn).
• Extend shared PythonCodegenBase to validate new task-specific inputs and to support additional payload/parse paths (including audio/media-gen paths present in this stacked diff).
• Harden JWT claim extraction by returning Optional.empty() when required custom claims are missing, plus add regression tests.

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/huggingFace/HuggingFaceInferenceOpDescSpec.scala	Adds dispatcher/codegen coverage tests for QA/ranking (and stacked audio/media-gen cases).
common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/huggingFace/HuggingFaceInferenceOpDesc.scala	Adds new operator JSON fields and registers new task-family codegens in the dispatcher.
common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/huggingFace/codegen/TaskCodegen.scala	Extends CodegenContext with QA/ranking/audio-related fields.
common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/huggingFace/codegen/QaRankingCodegen.scala	Implements payload + response parsing snippets for QA/ranking/classification tasks.
common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/huggingFace/codegen/PythonCodegenBase.scala	Extends shared Python template with validation, audio/media handling, and response normalization.
common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/huggingFace/codegen/MediaGenCodegen.scala	Adds task-family codegen for prompt-driven media generation parsing.
common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/huggingFace/codegen/AudioTaskCodegen.scala	Adds task-family codegen for audio tasks (raw-binary + JSON TTS).
common/auth/src/test/scala/org/apache/texera/auth/JwtParserSpec.scala	Adds tests ensuring missing required claims produce empty auth results.
common/auth/src/main/scala/org/apache/texera/auth/JwtParser.scala	Introduces claimsToOptionalSessionUser and uses it from parseToken.
amber/src/main/scala/org/apache/texera/web/auth/UserAuthenticator.scala	Switches to optional-claims parsing for Dropwizard auth adapter.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Ma77Ball]

Overall LGTM. Added small nit below.

[github-actions]

Automated Reviewer Suggestions

Based on the git blame history of the changed files, we recommend the following reviewers:

• No candidates found from git blame history.

[xuang7]

LGTM!