Discourse - #45

Open

jbaicoianu wants to merge 27 commits into

CarperAI:workingfrom

jbaicoianu:discourse

Commits on Oct 4, 2022

Initial commit of Discourse crawler and sample data
jbaicoianu
committed
1309c5e
Removed debug prints
jbaicoianu
committed
6a76f5b

Commits on Oct 5, 2022

Updated crawl site list
jbaicoianu
committed
79feea3
- Use target_dir for crawl output
jbaicoianu
committed
229f19c
Added "summary" and "topics" crawl types
jbaicoianu
committed
ea40840

Commits on Oct 10, 2022

Crawl list cleanup
jbaicoianu
committed
c5a247c
Profiler support, set retry option based on crawl type
jbaicoianu
committed
b9f6b8a
Improved crawl data structure, initial support for writing direct to tarballs, code parallelism and optimizations
jbaicoianu
committed
3aab776

Commits on Oct 19, 2022

Commits on Nov 2, 2022

Fix for problematic URL structures, improved category caching, first-pass attempt at verify function
jbaicoianu
committed
5f098ed

Commits on Nov 4, 2022

Reverted change to main codepile.py file
jbaicoianu
committed
ddecfc5
Whitespace
jbaicoianu
committed
3c6dd02

Commits on Nov 8, 2022

First pass at discourse preprocessor to convert raw json into lm_dataformat jsonl
jbaicoianu
committed
9d414fa
Added processor support, added command line options
jbaicoianu
committed
dc026ce
Updated dependencies
jbaicoianu
committed
6072e64

Commits on Nov 9, 2022

Big improvements to pipeline flow, with examples
jbaicoianu
committed
5ef703a
Improved usage examples
jbaicoianu
committed
73083fe
Use temp sitepath for processing
jbaicoianu
committed
50c77c1

Commits on Nov 11, 2022

Improved topic processor output
jbaicoianu
committed
c123b55

Commits on Nov 12, 2022

Improved subdir handling during processing
jbaicoianu
committed
aeca218
Improved path handling, log missing posts and exceptions
jbaicoianu
committed
344d5c5
Handle sites with missing topic slugs
jbaicoianu
committed
10de5f1

Commits on Nov 18, 2022

Improved metadata, support for paginated topics
jbaicoianu
committed
419e48a

Commits on Dec 16, 2022

Disable telnet server
jbaicoianu
committed
1dbbca3
Pass site parameter to get_additional_posts function
jbaicoianu
committed
438e11d

Commits on Feb 17, 2023

Handle licenses, use jsonl files instead of loose files
jbaicoianu
committed
145e77e