twintproject /twint
An advanced Twitter scraping & OSINT tool written in Python that doesn't use Twitter's API, allowing you to scrape a user's followers, following, Tweets and more while evading most API limitations.
★ 16,411⑂ 2,722👁 4⑃ 248● 1,175▤ 9
+1
日增长
16,411
stars
2,722
forks
4
watchers
248
prs
1,175
issues
9
releases
README
TWINT - Twitter Intelligence Tool
!2
!3
!
No authentication. No API. No limits.
Twint is an advanced Twitter scraping tool written in Python that allows for scraping Tweets from Twitter profiles without using Twitter's API.
Twint utilizes Twitter's search operators to let you scrape Tweets from specific users, scrape Tweets relating to certain topics, hashtags & trends, or sort out sensitive information from Tweets like e-mail and phone numbers. I find this very useful, and you can get really creative with it too.
Twint also makes special queries to Twitter allowing you to also scrape a Twitter user's followers, Tweets a user has liked, and who they follow without any authentication, API, Selenium, or browser emulation.
tl;dr Benefits
Some of the benefits of using Twint vs Twitter API:
- Can fetch almost __all__ Tweets (Twitter API limits to last 3200 Tweets only);
- Fast initial setup;
- Can be used anonymously and without Twitter sign up;
- No rate limitations.
Limits imposed by Twitter
Twitter limits scrolls while browsing the user timeline. This means that with .Profile or with .Favorites you will be able to get ~3200 tweets.
Requirements
- Python 3.6;
- aiohttp;
- aiodns;
- beautifulsoup4;
- cchardet;
- dataclasses
- elasticsearch;
- pysocks;
- pandas (=0.23.0);
- aiohttp_socks;
- schedule;
- geopy;
- fake-useragent;
- py-googletransx.
Installing
Git:
bash
git clone --depth=1 https://github.com/twintproject/twint.git
cd twint
pip3 install . -r requirements.txt
Pip:
bash
pip3 install twint
or
bash
pip3 install --user --upgrade git+https://github.com/twintproject/twint.git@origin/masteregg=twint
Pipenv:
bash
pipenv install git+https://github.com/twintproject/twint.gitegg=twint
March 2, 2021 Update
Added: Dockerfile
Noticed a lot of people are having issues installing (including me). Please use the Dockerfile temporarily while I look into them.
CLI Basic Examples and Combos
A few simple examples to help you understand the basics:
- twint -u username - Scrape all the Tweets of a user (doesn't include retweets but includes replies).
- twint -u username -s pineapple - Scrape all Tweets from the user's timeline containing _pineapple_.
- twint -s pineapple - Collect every Tweet containing pineapple from everyone's Tweets.
- twint -u username --year 2014 - Collect Tweets that were tweeted before 2014.
- twint -u username --since "2015-12-20 20:30:15" - Collect Tweets that were tweeted since 2015-12-20 20:30:15.
- twint -u username --since 2015-12-20 - Collect Tweets that were tweeted since 2015-12-20 00:00:00.
- twint -u username -o file.txt - Scrape Tweets and save to file.txt.
- twint -u username -o file.csv --csv - Scrape Tweets and save as a csv file.
- twint -u username --email --phone - Show Tweets that might have phone numbers or email addresses.
- twint -s "Donald Trump" --verified - Display Tweets by verified users that Tweeted about Donald Trump.
- twint -g="48.880048,2.385939,1km" -o file.csv --csv - STop issues
#604[ERROR] CRITICAL:twint.run:Twint:Feed:noDataExpecting ~ Inconsistent results [High Severity]148 comments · 2019-12-12#915CRITICAL:root:twint.run:Twint:Feed:noDataExpecting value: line 1 column 1 (char 0)96 comments · 2020-09-18#786[REQUEST] twint.get:User:'NoneType' object is not subscriptable58 comments · 2020-06-04#1266Twint not fetching Beyond 22 august 202151 comments · 2021-08-30#1061Error: twint.token.RefreshTokenException: Could not find the Guest token in HTML51 comments · 2020-12-07#863"Cannot find twitter account with name = ", This happened suddenly today, but it was fine yesterday.41 comments · 2020-08-12#958Timeout problem after API fix on Debian40 comments · 2020-10-12#918Twint stopped working out of the blue31 comments · 2020-09-19
Latest release
Fixes and little improvements. Changed Elasticsearch Schema v2.1.4 2019-10-19Improvements - since/until accept hh:mm:ss to fine the search - normalizer for cashtags,hashtags,mentions and usernames Thanks to @AFPMedialab - added support for Python3.8 Thanks to @Jamim