Skip to content

feat(frontend): add task-aware field visibility and preview to HuggingFace property editor#5568

Open
ELin2025 wants to merge 15 commits into
apache:mainfrom
ELin2025:hf/07-property-editor
Open

feat(frontend): add task-aware field visibility and preview to HuggingFace property editor#5568
ELin2025 wants to merge 15 commits into
apache:mainfrom
ELin2025:hf/07-property-editor

Conversation

@ELin2025

@ELin2025 ELin2025 commented Jun 8, 2026

Copy link
Copy Markdown
Contributor

What changes were proposed in this PR?

Wire up the HuggingFace operator's property editor so that selecting a task dynamically controls which fields are visible and shows a media preview card. This is the PR that makes the formly components from PRs #5566/ #5567 user-visible by mapping operator fields to custom field types in jsonSchemaMapIntercept.

Changes to operator-property-edit-frame.component.ts:

  • Map modelIdhuggingface formly type, imageInputhuggingface-image-upload, audioInputhuggingface-audio-upload
  • Hide the task field (it is controlled by the HuggingFaceComponent's task dropdown instead)
  • ~13 field visibility rules via formly expressions that show/hide fields based on the selected task (e.g., imageInput only for image tasks, contextColumn only for question-answering, systemPrompt/maxNewTokens/temperature only for text-generation)
  • 3 custom validators: requiredImageInput, requiredAudioInput, requiredPromptColumn — each checks whether the direct input or the corresponding column selector is filled
  • Removed validation: { show: true } from imageInput, audioInput, and promptColumn so errors only appear after the field is touched, not on a freshly-added operator
  • Added image-to-video to imageInputTasks so the image input field appears for that task
  • Task preview cards with sample media for 22 task types across 4 kinds (image, video, audio, text), plus a fallback for unknown tasks (sample-video.mp4 and sample-audio.wav are self-created)
  • Operator description truncation — clamps to 3 lines with CSS and adds a tooltip for the full text on hover

Changes to test infrastructure:

  • mockHuggingFaceSchema in mock-operator-metadata.data.ts (added to mockOperatorSchemaList)
  • mockHuggingFacePredicate in mock-workflow-data.ts

Any related issues, documentation, discussions?

How was this PR tested?

operator-property-edit-frame.component.spec.ts — 123 new tests

  • Task preview: all 20+ tasks (text-generation, image-classification, text-to-speech, text-to-video, ASR, image-to-image, object-detection, image-segmentation, image-to-text, document-question-answering, zero-shot-image-classification, image-text-to-text, sentence-similarity, text-ranking, translation, summarization, text-classification, token-classification, table-question-answering, feature-extraction, fill-mask), unknown task fallback, empty/whitespace task, non-HF operator, deleted operator guard, preview content validation (assetSrc, pills, outputBody)
  • Field type assignments: modelIdhuggingface, imageInputhuggingface-image-upload, audioInputhuggingface-audio-upload
  • Field visibility: imageInput, audioInput, inputImageColumn, inputAudioColumn, promptColumn, contextColumn, candidateLabels, sentencesColumn, systemPrompt, maxNewTokens, temperature — tested across 15+ tasks including mixed-input tasks (visual-question-answering, document-question-answering, image-text-to-text) and media-generation tasks (text-to-image, text-to-speech)
  • Undefined task visibility: all 11 fields hide when no task is selected
  • Validators: requiredPromptColumn, requiredImageInput, requiredAudioInput — pass/fail for empty, whitespace, model values, formControl values, inputColumn alternatives, zero-shot-image-classification edge cases, error message strings
  • getSelectedTask fallback: model.task → form.get → formControl.parent chain
  • Operator description: tooltip rendering, hidden when unset
  • Task field hidden for HF operators

hugging-face-image-upload.component.spec.ts — 27 new tests

  • Successful upload: happy path (fileName, formControl, model, dirty/touched), clears prior error
  • Compression pipeline: FileReader error, ArrayBuffer result, null result, Image onerror, null canvas context, quality reduction, dimension reduction, never fits, non-image toDataURL, 1x1 pixel, extreme aspect ratio
  • clearImage edge cases: non-string key guard, dirty/touched
  • Consecutive uploads: second upload replaces first

hugging-face.component.spec.ts — 11 new tests

  • Pagination edge cases: prevPage at page 0, goToPage on empty list
  • Task state snapshots: defaults on first visit to new task, preserve across switches
  • Server search edge cases: error handling, query superseding
  • getCurrentTaskTag / model selection / retryLoad edge cases

hugging-face-audio-upload.component.spec.ts — 7 new tests

  • formControl state: dirty/touched after upload and clear
  • Model update: non-string key skips model write
  • previewSrc edge cases: non-audio data URL, alternate audio MIME type
  • Consecutive uploads: replacement
  • revokePreviewUrl: no throw without preview

Run with ng test.

Was this PR authored or co-authored using generative AI tooling?

Co-authored with Claude Opus 4.6

@github-actions github-actions Bot added frontend Changes related to the frontend GUI common labels Jun 8, 2026
@codecov-commenter

codecov-commenter commented Jun 8, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 65.13158% with 53 lines in your changes missing coverage. Please review.
✅ Project coverage is 57.34%. Comparing base (021818d) to head (27e98e5).

Files with missing lines Patch % Lines
...-frame/operator-property-edit-frame.component.html 3.84% 50 Missing ⚠️
...it-frame/operator-property-edit-frame.component.ts 96.93% 1 Missing and 2 partials ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##               main    #5568      +/-   ##
============================================
+ Coverage     57.17%   57.34%   +0.16%     
- Complexity     3100     3101       +1     
============================================
  Files          1130     1130              
  Lines         43825    43978     +153     
  Branches       4747     4790      +43     
============================================
+ Hits          25056    25218     +162     
+ Misses        17330    17321       -9     
  Partials       1439     1439              
Flag Coverage Δ *Carryforward flag
access-control-service 70.00% <ø> (ø)
agent-service 44.59% <ø> (ø) Carriedforward from 82880b8
amber 59.47% <ø> (+0.01%) ⬆️
computing-unit-managing-service 0.00% <ø> (ø)
config-service 52.30% <ø> (ø)
file-service 62.81% <ø> (ø)
frontend 50.51% <65.13%> (+0.45%) ⬆️
notebook-migration-service 78.57% <ø> (ø)
pyamber 91.15% <ø> (ø) Carriedforward from 82880b8
python 90.69% <ø> (ø) Carriedforward from 82880b8
workflow-compiling-service 55.14% <ø> (ø)

*This pull request uses carry forward flags. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@ELin2025

ELin2025 commented Jun 8, 2026

Copy link
Copy Markdown
Contributor Author

/request-review @Ma77Ball

@Ma77Ball Ma77Ball left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left comments for review below.

@ELin2025 ELin2025 force-pushed the hf/07-property-editor branch from 553d90d to d1dc52a Compare June 11, 2026 23:38
@github-actions

github-actions Bot commented Jun 11, 2026

Copy link
Copy Markdown
Contributor

⚠️ Benchmark changes need a look

🟢 0 better · 🔴 9 worse · ⚪ 6 noise (<±5%) · 0 without baseline

Compared against main 021818d benchmarked on this same runner, so the delta is largely free of cross-runner hardware noise. The "7d avg" column still reflects the gh-pages dashboard. Treat <±5% as noise unless repeated.

Dashboard · Run

config throughput MB/s latency max Δ latest / 7d
🔴 bs=10 sw=10 sl=64 399 0.244 23,933/38,964/38,964 us 🔴 +25.6% / 🔴 +154.9%
🔴 bs=100 sw=10 sl=64 800 0.488 117,670/168,399/168,399 us 🔴 +22.5% / 🔴 +55.4%
bs=1000 sw=10 sl=64 944 0.576 1,059,301/1,111,267/1,111,267 us ⚪ within ±5% / 🔴 +7.2%
Baseline details

Latest main 021818d from same runner

config metric PR latest main 7d avg Δ latest Δ 7d
bs=10 sw=10 sl=64 throughput 399 tuples/sec 467 tuples/sec 770.95 tuples/sec -14.6% -48.2%
bs=10 sw=10 sl=64 MB/s 0.244 MB/s 0.285 MB/s 0.471 MB/s -14.4% -48.1%
bs=10 sw=10 sl=64 p50 23,933 us 19,819 us 12,775 us +20.8% +87.3%
bs=10 sw=10 sl=64 p95 38,964 us 31,013 us 15,286 us +25.6% +154.9%
bs=10 sw=10 sl=64 p99 38,964 us 31,013 us 18,795 us +25.6% +107.3%
bs=100 sw=10 sl=64 throughput 800 tuples/sec 873 tuples/sec 976.93 tuples/sec -8.4% -18.1%
bs=100 sw=10 sl=64 MB/s 0.488 MB/s 0.533 MB/s 0.596 MB/s -8.4% -18.2%
bs=100 sw=10 sl=64 p50 117,670 us 113,408 us 102,557 us +3.8% +14.7%
bs=100 sw=10 sl=64 p95 168,399 us 137,465 us 108,383 us +22.5% +55.4%
bs=100 sw=10 sl=64 p99 168,399 us 137,465 us 115,249 us +22.5% +46.1%
bs=1000 sw=10 sl=64 throughput 944 tuples/sec 967 tuples/sec 1,009 tuples/sec -2.4% -6.4%
bs=1000 sw=10 sl=64 MB/s 0.576 MB/s 0.59 MB/s 0.616 MB/s -2.4% -6.5%
bs=1000 sw=10 sl=64 p50 1,059,301 us 1,032,971 us 997,695 us +2.5% +6.2%
bs=1000 sw=10 sl=64 p95 1,111,267 us 1,078,185 us 1,036,731 us +3.1% +7.2%
bs=1000 sw=10 sl=64 p99 1,111,267 us 1,078,185 us 1,069,334 us +3.1% +3.9%
Raw CSV
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,500.66,200,128000,399,0.244,23933.32,38964.17,38964.17
1,100,10,64,20,2499.44,2000,1280000,800,0.488,117669.81,168398.69,168398.69
2,1000,10,64,20,21183.86,20000,12800000,944,0.576,1059301.20,1111267.30,1111267.30

@ELin2025 ELin2025 force-pushed the hf/07-property-editor branch 3 times, most recently from eafce6d to c009c34 Compare June 12, 2026 00:09
@ELin2025 ELin2025 requested a review from Ma77Ball June 15, 2026 00:52
@ELin2025 ELin2025 force-pushed the hf/07-property-editor branch 2 times, most recently from 7d88b62 to 1bf97cb Compare June 16, 2026 18:08
@github-actions github-actions Bot added engine dependencies Pull requests that update a dependency file pyamber ci changes related to CI docs Changes related to documentations infra platform Non-amber Scala service paths agent-service labels Jun 16, 2026
@ELin2025 ELin2025 force-pushed the hf/07-property-editor branch from 1bf97cb to 5f3afac Compare June 16, 2026 18:14
@github-actions github-actions Bot removed engine dependencies Pull requests that update a dependency file pyamber ci changes related to CI docs Changes related to documentations infra labels Jun 16, 2026

@xuang7 xuang7 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall, this looks good to me. Please update the target branch to the latest main and rebase your branch, as the current diff includes some unrelated changes. Thanks.

@ELin2025 ELin2025 force-pushed the hf/07-property-editor branch 8 times, most recently from 600ba81 to 319fbe4 Compare June 30, 2026 17:10
@ELin2025

Copy link
Copy Markdown
Contributor Author

@xuang7 branch has been rebased

@xuang7 xuang7 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall looks good!
sample-audio.wav (170 KB) and sample-video.mp4 (1.25 MB) are new binary files. Could you confirm whether they are self-created or under an ASF-compatible license, and note the source in the PR? Also, the .mp4 is relatively large for a preview thumbnail, so compressing or shortening it would be nice.

ELin2025 and others added 12 commits July 2, 2026 10:03
…gFace property editor

Show/hide operator fields based on the selected HuggingFace task (e.g.,
imageInput only for image tasks, contextColumn only for question-answering).
Adds task preview cards with media samples per task kind (image/video/audio/text),
custom validators for required inputs, and ~13 field visibility rules inside
the formly jsonSchemaMapIntercept.

Co-Authored-By: Anish Shivamurthy <anish@uci.edu>
Add mockHuggingFaceSchema and mockHuggingFacePredicate to test infrastructure.
Add 7 spec tests covering huggingFaceTaskPreview for known tasks (text, image,
audio, video), unknown tasks (fallback), empty tasks, and non-HF operators.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…ew sample

Fix inverted fallback for systemPrompt/maxNewTokens/temperature: these
fields now correctly hide when no task is selected, matching the behavior
of all other HuggingFace fields. Add missing image-text-to-text entry to
huggingFaceTaskPreviewSamples so it no longer falls through to the
generic text fallback.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Add sample-video.mp4 and sample-audio.wav for task preview cards
- Check hasOperator() before getOperator() in isHuggingFaceOperator()

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…lassification

Prefer self.CANDIDATE_LABELS over prompt column for candidate labels,
with fallback to prompt_value for backward compatibility.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Cover hide expressions for imageInput, audioInput, promptColumn,
systemPrompt, contextColumn, and candidateLabels across task types.
Also test requiredPromptColumn, requiredImageInput, and
requiredAudioInput validators pass/fail for relevant tasks.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…iew tests

Add tests for sentencesColumn, inputImageColumn, inputAudioColumn
visibility, validator edge cases with column fallbacks, and additional
task preview kinds.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…ation show:true

Add missing image-to-video task to imageInputTasks so the image input
field appears for that task. Remove validation: { show: true } from
imageInput, audioInput, and promptColumn so errors only appear after
the field is touched, not on a freshly-added operator.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…fields

Add tests for field type assignments, undefined task visibility,
image-to-video/image-to-image/document-question-answering tasks,
audio-classification, prompt column visibility for mixed tasks,
validator formControl value paths, additional task previews, and
deleted operator guard.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@ELin2025 ELin2025 force-pushed the hf/07-property-editor branch from b4769c8 to 191073b Compare July 2, 2026 17:03
@ELin2025

ELin2025 commented Jul 2, 2026

Copy link
Copy Markdown
Contributor Author

@xuang7 mp4 has been compressed to 116 KB. Both mp4 and wav file were self-created and noted in the PR

@Yicong-Huang

Copy link
Copy Markdown
Contributor

Hi @ELin2025, thanks for the PR. Two suggestions:

  • for stacked PR, the note about stack can become obslete when referened PR get merged. Please keep that note updated, or remove it once it is ready for review
    ⚠️ This PR is stacked on https://github.com/apache/texera/pull/5567. Until that lands, the diff below may also include PR 6a/6b's task selector, model browser, and audio upload changes depending on which base GitHub is showing.
  • can we make the coverage higher? it can help us prevent regression of the feature in the future.
    Thanks!

@ELin2025

ELin2025 commented Jul 3, 2026

Copy link
Copy Markdown
Contributor Author

@xuang7 coverage increased and PR comment edited!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

common frontend Changes related to the frontend GUI

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add task-aware field visibility and preview to HuggingFace property editor

6 participants