sebastianruder--nlp-progress
rstojnic
3bc764673f
* First complete version of the parser * First complete run, now just need to fix the edge cases. * Rename the folder, better handling of tables * Handling of subdatasets * Make this file consistent with how subdatasets are specified in other markdown files. * Extract all the links in the dataset description and put them into a structured format * tweak wording * Missing comments * Add attribution * Model name now only extracts the model name - not the author tags as well
Exporting NLP-progress into a structure format
Parse and export the unstructured information from Markdown into a structured JSON format.
Installation
Requires Python 3.6+.
Create a virtualenv and install requirements (you can also use conda):
virtualenv -p python3 venv
source venv/bin/activate
pip install -r requirements.txt
Running
From the NLP-progress root directly (where the LICENCE file is), run:
python structured/export.py <one or more directories or files>
For example, to export all the data in the english/ directory:
python structured/export.py english
By default the output will be written into structured.json, but you can override this with the --output parameter.