Skip to content

Warn users about silent dataset downloads and show live progress#1040

Open
zhenchaoni wants to merge 1 commit into
mainfrom
private/zhenni/download_dataset_warning
Open

Warn users about silent dataset downloads and show live progress#1040
zhenchaoni wants to merge 1 commit into
mainfrom
private/zhenni/download_dataset_warning

Conversation

@zhenchaoni

Copy link
Copy Markdown
Member

Fix #444

Problem

Large HuggingFace dataset downloads are triggered silently inside blocking quantize/eval stages. The UI would freeze for minutes with only a static spinner while GB of data downloaded in the background — users couldn't tell whether the tool was working or hung (#444).

Changes

Surface dataset loads as warnings
Dataset-loading logs are promoted from infowarning at every load_dataset entry point, so users get a visible heads-up before a potentially large fetch begins:

  • datasets/image.py, datasets/image_segmentation.py, datasets/text.py
  • eval/base_evaluator.py
  • Added an equivalent warning to the Qwen3 GSM8K calibration loader (quant/calibration/qwen3_transformer_only.py)

Live elapsed-time counter during blocking stages
StageLive (utils/console.py) now renders a self-updating elapsed-seconds counter on Rich's refresh thread (15 fps), so the timer keeps ticking even while the main thread is blocked in native code (downloads, quantization). Added _t0/_done tracking so __exit__ finalizes timing if a stage exits without an explicit set_done.

Clearer eval messaging
Split the eval console output — Loading dataset... is now distinct from Evaluating..., so the download phase is attributable instead of hidden behind a combined "Loading dataset and evaluating..." message.

Suppress cosmetic noise
Filter the scikit-learn "number of unique classes is greater than 50% of the number of samples" UserWarning (_warnings.py), which fires from HF evaluate metrics on small eval sample counts and would otherwise clutter the newly surfaced output.

Result

Users now see an upfront warning when a dataset is about to load and a continuously updating timer during long stages, resolving the "frozen with no feedback" experience described in #444.

@zhenchaoni zhenchaoni requested a review from a team as a code owner July 3, 2026 07:30

@DingmaomaoBJTU DingmaomaoBJTU left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice improvement to the "frozen with no feedback" UX — the live elapsed timer and the upfront dataset heads-up are both welcome. A few things worth a look before merge: one likely regression in StageLive.__exit__ (failures can render as a green ✓), the eval message ordering, and the log-level semantics of the info→warning promotion. Details inline.

Comment on lines +376 to +377
if not self._done:
self.set_done(time.monotonic() - self._t0)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This finalizer throws away the exception state (*_), so a stage that raises without first calling set_error now gets finalized as a green ✓ with an elapsed time — printed immediately before the traceback.

Concrete paths that hit this today:

  • The entire export stage (build.py ~1398–1422): if _load_model/export_onnx raise, nothing calls set_error, so the failed stage renders as done ✓.
  • compile stage (build.py ~1307): raise RuntimeError("Compile reported success but output not found...") runs with no set_error.
  • quantize stage (build.py ~1213): if quantize_onnx itself raises (rather than returning success=False), no set_error.

Since every success path already calls set_done, this if not self._done branch in practice only fires on uncaught exceptions — i.e. it paints failures green. Suggest keying off the exception info instead:

def __exit__(self, exc_type, exc_val, exc_tb) -> None:
    if not self._done:
        if exc_type is not None:
            self.set_error()
        else:
            self.set_done(time.monotonic() - self._t0)
    if self._live:
        ...

cls = get_evaluator_class(config)
try:
console.print("[bold]Loading dataset and evaluating...[/bold]")
console.print("[bold]Evaluating...[/bold]")

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The ordering reads backwards now: this prints Evaluating..., but the dataset load actually happens on the next line inside cls(config, model)WinMLEvaluator.__init__prepare_data(). So the user sees Evaluating... first and then the Loading dataset: warning underneath it. Given the PR's goal is to make the download phase attributable, consider printing a Loading dataset... line before constructing the evaluator (or moving this Evaluating... print to just before task_evaluator.compute()).

# bulk download.
streaming = self._config.get("streaming", False) and self._max_samples is not None
logger.info(f"Loading dataset: {self._dataset_name} with split: {self._data_split}")
logger.warning(f"Loading dataset: {self._dataset_name} with split: {self._data_split}")

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Promoting normal, successful status to warning is a bit of a log-level smell — these are informational messages, not warnings. It only surfaces because the default level is WARNING (INFO is hidden), but it means every successful run emits WARNING records that CI / log processing may flag, and it diverges from how user-facing status is surfaced elsewhere via console.print. Same applies to the other promoted sites (image_segmentation.py, text.py, base_evaluator.py, qwen3_transformer_only.py).

Since these loaders are deep library code with no console handle, warning is a pragmatic call — but worth a code comment or a follow-up toward a proper user-facing status channel so log semantics stay clean.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Warn before large downloads in long-running commands (build, quantize) + live elapsed-time counter

2 participants