Social Media Intelligence
Tweet collection
A script that can be used to collect tweets on a specific query. The script reconnects every [duration] minutes and uses a document from a collection of queries as input. Based on the language of the query, a different developer account is used.
Example usage:
- nohup python3 collectscript.py -duration 60 -directory /home/vagrant/logs/ -filename nlcollectlog.log -db SOCMINT -tcol streamtest -qcol twitter_queries &
- nohup python3 /home/vagrant/socmint/collectscript.py -duration 60 -filename frcollectlog.log -db SOCMINT -tcol frenchtest -qcol twitter_queries_FR &
kwargs:
- directory: The directory you want to use. If none is specified this defaults to ~/logs/. The directory will be created if it does not exist. Requires absolute path.
- filename: The filename you want to use. If none is specified this defaults to collectlog.txt.
- duration: Duration of the data collection before reconnecting expressed in minutes, defaults to 60.
- db: Database for storage on Mongoserver. default="SOCMINT".
- tcol: Collection for tweet storage in Database. default="streamtest".
- qcol: Collection for stream API queries in Database. default="twitter_queries".
Tweet analysis
A script that can be used to update the query based on a time window. The script takes a time window of tweets, and extracts trending hashtags. Only tweets in the same language as the query are used to realise this (an index was created for this on the 'created_at' and the 'lang' fields of the data to speed up the search).
Example usage:
- python3 /home/vagrant/socmint/queryupdatescript.py -period 4 -directory /home/vagrant/logs/ -filename queryupdatelog.log -debug True -base False -db SOCMINT -tcol streamtest -qcol twitter_queries
- python3 /home/vagrant/socmint/queryupdatescript.py -period 4 -directory /home/vagrant/logs/ -filename frqueryupdatelog.log -debug True -base False -db SOCMINT -tcol frenchtest -qcol twitter_queries_FR
kwargs:
- directory: The directory you want to use. If none is specified this defaults to ~/logs/. The directory will be created if it does not exist. Requires absolute path.
- filename: The filename you want to use. If none is specified this defaults to querylog.txt.
- period: windows width in hours, defaults to 4.
- db: Database for storage on Mongoserver. default="SOCMINT".
- tcol: Collection for tweet storage in Database. default="streamtest".
- qcol: Collection for stream API queries in Database. default="twitter_queries".
- debug: bolean for debug mode (extra info in log file).
- base: bolean that imposes base query as reference.
Data collection server
Server running the data collection queries for the SOCMINT project.
Production server: 172.20.101.18
- Ubuntu 16.04.5 LTS
- MongoDB v4.0.6
- Python 3.5.2 for data collection via stream API
- Two Twitter developper accounts
Check this project on GitLab