This project intercepts/manipulates/requests/handles data. Then, it does one of many things:
- Formats the data
- Stores the data in a neo4j database.
- Stores the data in a mongo database.
- Stores the data in a different datastore entirely.
- Kicks off an ETL process (planned Mid Oct 2016)
- Etc.
This app is planned to be a gate keeper or router-like microservice for the rest of the engines, such as a machine learning engine (Non-existent in 2016), NLP engine ('speedy-affect-scorer' in 2016), ETL engine ('energetic-etl') or another engine.
Its primary function is getting data from its counterpart in the front end side of the Logro application (currently callled 'fixed-gateway' in 2016). Its other important function is sending data to its counterpart engines listed above via its mechanisms and API.
The project originally was designed to also do ETL functions which are now part of 'energetic-etl' (GPLv3) where the intercepts directory contained some of the logic this project used as boilerplate. See the other project called: energetic-etl
Find the project here: https://github.com/nbuechler/hungry-interceptor
- First, install virtualenv if not done so already -- https://virtualenv.pypa.io/en/latest/installation.html(https://virtualenv.pypa.io/en/latest/installation.html)
- Then, run this command:
$ virtualenv venv
- Next, activate the virtual environment (make sure you get the'.'):
$ . venv/bin/activate
- Last, install the requirements with pip:
$ pip install -r requirements.txt
I've begun working with a friend of mine in early 2016, and have found the usefulness in planning to intercept the data and send it to a machine learning model to produce similar results. This machine learning model is in a separate project and will likely be linked here when the project is in a stable state.
- Scaling the application to work in a cluster will be challenging due to the single-threaded nature of Flask.
- Considering a separate microservice specifically for coordinating message queues, akin to something like Celery or RabbitMQ would be useful (but, I'm not sure yet if this is viable or a good idea)
- It also might be valuable to rework this project to be more of a microservice, instead of a service-oriented project. Thus, looking into microservice architectures to apply to the entire Logro project is a good idea to consider.
From a terminal, start mongo:
mongod
From a terminal start Neo4j:
sudo /etc/init.d/neo4j-service start
python app/runserver.py 5000
This contains a port - 5000 - for running local, otherwise it will try to run on the default port - 80 - and that's taken. If you run it on a server instance, such as one on AWS without the port specified, it should running open to the world. That is the usual configuration for deploying code and making it 'live' to the world. But, make sure you are prudent in running the code in the way you want it to run.
The scripts folder is for automation. If using AWS, here's a problem I encountered in the middle of Feb, 2016: DO NOT UPGRADE PIP from 6.1.1 on a standard AWS EC2 instance, or it will bite you. I don't really know why so this might have changed when you are reading this.
- Flask==0.10.1
- Flask-Cors==2.1.0
- Flask-PyMongo==0.3.1
- Flask-WTF==0.11
- itsdangerous==0.24
- Jinja2==2.8
- MarkupSafe==0.23
- py2neo==2.0.8
- pymongo==2.8.1
- six==1.10.0
- Werkzeug==0.11.3
- wheel==0.24.0
- WTForms==2.0.2
- Refactor: Remove old code from 'hungry-interceptor' when this is complete!
Make sure to pay attention to how CORS right now accepts everything.
See more here: https://flask-cors.readthedocs.org/en/latest/
from flask.ext.cors import CORS
cors = CORS(app, resources={r"/\*": {"origins": "\*"}}) #CORS :WARNING everything!
GPLv3