Restructure#75
Conversation
* added a sentiment outlook to the dashboard * Changed the color pallete * added computation of the last sentiment percentile
* moved all controller to thereadingmachine/ * added path to thereadingmachine for python * collected all variables and parameters in parameter.py and environment.py
* added a sentiment outlook to the dashboard * Changed the color pallete * added computation of the last sentiment percentile
* moved all controller to thereadingmachine/ * added path to thereadingmachine for python * collected all variables and parameters in parameter.py and environment.py
This eliminates the manula update everytime we do testing and avoid resetting the database.
mrpozzi
left a comment
mrpozzi
left a comment
There was a problem hiding this comment.
Couple of comment, but all in all much cleaner, good job
| ``` | ||
|
|
||
| You can view the repo on [Docker Hub](https://hub.docker.com/r/thereadingmachine/thereadingmachine/) | ||
| You can view the repo on [Docker |
| export WEBAPP_DIR=$PROJECT_HOME/webapp | ||
| export WEBAPP_PLOT_DIR=$WEBAPP_DIR/templates/static/plotly | ||
| export PYTHONPATH=$PROJECT_HOME:$PYTHONPATH | ||
| export AIRFLOW_START_DATE=`date +%Y-%m-%d` |
There was a problem hiding this comment.
setting up the start date as today's date may have some unwanted effect: namely if you schedule tasks that take a long time the dag may never actually run. It's not encouraged practice in Airflow. Consider using a fixed start date or adding a buffer (like 1/2 days before today's date)
There was a problem hiding this comment.
- Not sure what you meant by it will never run. Why would the dag never run if a process takes a long time?
- It is true that the start date should not be dynamic in the
dagfile, however, theset_env_var.shfile is merely a configuration which can be changed. Generally, you will have multiple configuration file for testing, staging and deployment. This avoids having to change the dag file every single time we deploy, which I have done multiple time.
There was a problem hiding this comment.
Just a matter of code quality, besides having a variable parameter impacts reproducibility.
| @@ -3,55 +3,28 @@ | |||
| import pandas as pd | |||
There was a problem hiding this comment.
Didn't we decide to drop the lstm process?
There was a problem hiding this comment.
Dropped from the process (i.e. pipeline) but we don't have to drop it from the code.
3 participants
thereadingmachinedirectory.