karpathy /arxiv-sanity-preserver
Web interface for browsing, search and filtering recent arxiv submissions
★ 5,808⑂ 1,352👁 162⑃ 62● 117▤ 0
+1
日增长
5,808
stars
1,352
forks
162
watchers
62
prs
117
issues
0
releases
README
arxiv sanity preserver Update Nov 27, 2021: you may wish to look at my from-scratch re-write of arxiv-sanity: arxiv-sanity-lite. It is a smaller version of arxiv-sanity that focuses on the core value proposition, is significantly less likely to ever go down, scales better, and has a few additional goodies such as multiple possible tags per account, regular emails of new papers of interest, etc. It is also running live on arxiv-sanity-lite.com. This project is a web interface that attempts to tame the overwhelming flood of papers on Arxiv. It allows researchers to keep track of recent papers, search for papers, sort papers by similarity to any paper, see recent popular papers, to add papers to a personal library, and to get personalized recommendations of (new or old) Arxiv papers. This code is currently running live at www.arxiv-sanity.com/, where it's serving 25,000+ Arxiv papers from Machine Learning (cs.[CV|AI|CL|LG|NE]/stat.ML) over the last ~3 years. With this code base you could replicate the website to any of your favorite subsets of Arxiv by simply changing the categories in fetch_papers.py. !user interface Code layout There are two large parts of the code: Indexing code. Uses Arxiv API to download the most recent papers in any categories you like, and then downloads all papers, extracts all text, creates tfidf vectors based on the content of each paper. This code is therefore concerned with the backend scraping and computation: building up a database of arxiv papers, calculating content vectors, creating thumbnails, computing SVMs for people, etc. User interface. Then there is a web server (based on Flask/Tornado/sqlite) that allows searching through the database and filtering papers by similarity, etc. Dependencies Several: You will need numpy, feedparser (to process xml files), scikit learn (for tfidf vectorizer, training of SVM), flask (for serving the results), flask_limiter, and tornado (if you want to run the flask server in production). Also dateutil, and scipy. And sqlite3 for database (accounts, library support, etc.). Most of these are easy to get through pip, e.g.: bash $ virtualenv env optional: use virtualenv $ source env/bin/activate optional: use virtualenv $ pip install -r requirements.txt You will also need ImageMagick and pdftotext, which you can install on Ubuntu as sudo apt-get install imagemagick poppler-utils. Bleh, that's a lot of dependencies isn't it. Processing pipeline The processing pipeline requires you to run a series of scripts, and at this stage I really encourage you to manually inspect each script, as they may contain various inline settings you might want to change. In order, the processing pipeline is: 1. Run fetch_papers.py to query arxiv API and create a file db.p that contains all information for each paper. This script is where you would modify the query, indicating which parts of arxiv you'd like to use. Note that if you're trying to pull too many papers ar
Top issues
#57Recommended papers is empty18 comments · 2017-01-30#165Site is down11 comments · 2021-05-19#125Top hype not refreshing10 comments · 2019-03-18#52Feature Request: Expanding to all of arXiv10 comments · 2016-12-26#26Feature Requests/Suggestions10 comments · 2016-03-26#114What is the / are there better active fork(s) for this?8 comments · 2018-07-03#140Site refused to connect6 comments · 2020-05-05#7Any plans for adding feature of commenting on papers? 6 comments · 2015-12-05