Implement `stream` option in high-level api

[abetlen]

Allow tokens to be generated one at a time while still terminating before a stop sequence is emitted (may need to keep tokens buffered in the generator).