tools/build_vocab.lua

build_vocab.lua options:

  • -h [] (default: false)
    This help.
  • -md [] (default: false)
    Dump help in Markdown format.
  • -config (default: '')
    Load options from this file.
  • -save_config (default: '')
    Save options to this file.

Vocabulary options

  • -data (required)
    Data file.
  • -save_vocab (required)
    Vocabulary dictionary prefix.
  • -vocab_size (default: 50000)
    List of source vocabularies size: word[ feat1[ feat2[ ...] ] ]. If = 0, vocabularies are not pruned.
  • -words_min_frequency
  • (default: 0)
    List of source words min frequency: word[ feat1[ feat2[ ...] ] ]. If = 0, vocabularies are pruned by size.
  • -keep_frequency [] (default: false)
    Keep frequency of words in dictionary.
  • -idx_files [] (default: false)
    If set, each line of the data file starts with a first field which is the index of the sentence.
  • Logger options

    • -log_file (default: '')
      Output logs to a file under this path instead of stdout - if file name ending with json, output structure json.
    • -disable_logs [] (default: false)
      If set, output nothing.
    • -log_level (accepted: DEBUG, INFO, WARNING, ERROR, NOERROR; default: INFO)
      Output logs at this level and above.