Warn users about silent dataset downloads and show live progress#1040
Warn users about silent dataset downloads and show live progress#1040zhenchaoni wants to merge 1 commit into
Conversation
DingmaomaoBJTU
left a comment
There was a problem hiding this comment.
Nice improvement to the "frozen with no feedback" UX — the live elapsed timer and the upfront dataset heads-up are both welcome. A few things worth a look before merge: one likely regression in StageLive.__exit__ (failures can render as a green ✓), the eval message ordering, and the log-level semantics of the info→warning promotion. Details inline.
| if not self._done: | ||
| self.set_done(time.monotonic() - self._t0) |
There was a problem hiding this comment.
This finalizer throws away the exception state (*_), so a stage that raises without first calling set_error now gets finalized as a green ✓ with an elapsed time — printed immediately before the traceback.
Concrete paths that hit this today:
- The entire
exportstage (build.py~1398–1422): if_load_model/export_onnxraise, nothing callsset_error, so the failed stage renders as done ✓. compilestage (build.py~1307):raise RuntimeError("Compile reported success but output not found...")runs with noset_error.quantizestage (build.py~1213): ifquantize_onnxitself raises (rather than returningsuccess=False), noset_error.
Since every success path already calls set_done, this if not self._done branch in practice only fires on uncaught exceptions — i.e. it paints failures green. Suggest keying off the exception info instead:
def __exit__(self, exc_type, exc_val, exc_tb) -> None:
if not self._done:
if exc_type is not None:
self.set_error()
else:
self.set_done(time.monotonic() - self._t0)
if self._live:
...| cls = get_evaluator_class(config) | ||
| try: | ||
| console.print("[bold]Loading dataset and evaluating...[/bold]") | ||
| console.print("[bold]Evaluating...[/bold]") |
There was a problem hiding this comment.
The ordering reads backwards now: this prints Evaluating..., but the dataset load actually happens on the next line inside cls(config, model) → WinMLEvaluator.__init__ → prepare_data(). So the user sees Evaluating... first and then the Loading dataset: warning underneath it. Given the PR's goal is to make the download phase attributable, consider printing a Loading dataset... line before constructing the evaluator (or moving this Evaluating... print to just before task_evaluator.compute()).
| # bulk download. | ||
| streaming = self._config.get("streaming", False) and self._max_samples is not None | ||
| logger.info(f"Loading dataset: {self._dataset_name} with split: {self._data_split}") | ||
| logger.warning(f"Loading dataset: {self._dataset_name} with split: {self._data_split}") |
There was a problem hiding this comment.
Promoting normal, successful status to warning is a bit of a log-level smell — these are informational messages, not warnings. It only surfaces because the default level is WARNING (INFO is hidden), but it means every successful run emits WARNING records that CI / log processing may flag, and it diverges from how user-facing status is surfaced elsewhere via console.print. Same applies to the other promoted sites (image_segmentation.py, text.py, base_evaluator.py, qwen3_transformer_only.py).
Since these loaders are deep library code with no console handle, warning is a pragmatic call — but worth a code comment or a follow-up toward a proper user-facing status channel so log semantics stay clean.
Fix #444
Problem
Large HuggingFace dataset downloads are triggered silently inside blocking quantize/eval stages. The UI would freeze for minutes with only a static spinner while GB of data downloaded in the background — users couldn't tell whether the tool was working or hung (#444).
Changes
Surface dataset loads as warnings
Dataset-loading logs are promoted from
info→warningat everyload_datasetentry point, so users get a visible heads-up before a potentially large fetch begins:datasets/image.py,datasets/image_segmentation.py,datasets/text.pyeval/base_evaluator.pyquant/calibration/qwen3_transformer_only.py)Live elapsed-time counter during blocking stages
StageLive(utils/console.py) now renders a self-updating elapsed-seconds counter on Rich's refresh thread (15 fps), so the timer keeps ticking even while the main thread is blocked in native code (downloads, quantization). Added_t0/_donetracking so__exit__finalizes timing if a stage exits without an explicitset_done.Clearer eval messaging
Split the eval console output —
Loading dataset...is now distinct fromEvaluating..., so the download phase is attributable instead of hidden behind a combined "Loading dataset and evaluating..." message.Suppress cosmetic noise
Filter the scikit-learn "number of unique classes is greater than 50% of the number of samples"
UserWarning(_warnings.py), which fires from HFevaluatemetrics on small eval sample counts and would otherwise clutter the newly surfaced output.Result
Users now see an upfront warning when a dataset is about to load and a continuously updating timer during long stages, resolving the "frozen with no feedback" experience described in #444.