Skip to content

Documentation update#207

Merged
amercader merged 5 commits intomasterfrom
doc-reorg
Oct 7, 2020
Merged

Documentation update#207
amercader merged 5 commits intomasterfrom
doc-reorg

Conversation

@amercader
Copy link
Copy Markdown
Member

@amercader amercader commented Oct 5, 2020

Fixes #198
Fixes #200
Fixes #129

  • Updated and simplified documentation, consolidated in the README
  • Document all config options and defaults, add support for defining them via env vars
  • Document concurrent setup with Postgres + uWSGI threads

@amercader amercader marked this pull request as ready for review October 5, 2020 18:50
@amercader
Copy link
Copy Markdown
Member Author

@mbocevski @jqnatividad would love your input on this, particularly on the High Availability Setup. @jqnatividad I think I captured all your proposed changes but let me know if you think something is missing.

Comment thread README.md
In order to tell CKAN where this webservice is located, the following must be
added to the `[app:main]` section of your CKAN configuration file :

ckan.datapusher.url = http://0.0.0.0:8800/
Copy link
Copy Markdown
Contributor

@mbocevski mbocevski Oct 6, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's better the default URL to be http://127.0.0.1:8800 as is in the config files.

Suggested change
ckan.datapusher.url = http://0.0.0.0:8800/
ckan.datapusher.url = http://127.0.0.1:8800/

Comment thread README.md
Comment on lines +65 to +66
If you need to change the host or port, copy `deployment/datapusher_settings.py` to
`deployment/datapusher_local_settings.py` and modify the file to suit your needs.
Copy link
Copy Markdown
Contributor

@mbocevski mbocevski Oct 6, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When changing the HOST and PORT of datapusher in settings, one must also change them in the uwsgi config as well https://github.com/ckan/datapusher/blob/master/deployment/datapusher-uwsgi.ini#L3.

Suggested change
If you need to change the host or port, copy `deployment/datapusher_settings.py` to
`deployment/datapusher_local_settings.py` and modify the file to suit your needs.
If you need to change the host or port, copy `deployment/datapusher_settings.py` to
`deployment/datapusher_local_settings.py` and modify the file to suit your needs.
Make sure to update http setting in `deployment/datapusher-uwsgi.ini` to match your host:port
setting.

@mbocevski
Copy link
Copy Markdown
Contributor

@amercader Looks clean and I like moving the docs to the repo itself, I had 2 minor comments. Perhaps it's a good idea to include a section in the README about the uwsgi settings, cause we also are setting uid and gid to www-data there and depending on the system, this user/group combination might not exist. It's enough if we point to users to review the uwsgi config to make sure that it suits their system, prior to starting it.

@amercader
Copy link
Copy Markdown
Member Author

Thanks @mbocevski I added all your suggestions

Comment thread README.md
one you need to adapt the relevant paths in the `datapusher-uwsgi.ini` to the ones you are using.


### High Availability Setup
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great! This section captures the essence of what I did in the two PRs.

Comment thread README.md
| -- | -- | -- |
| HOST | '0.0.0.0' | Web server host |
| PORT | 8800 | Web server port |
| SQLALCHEMY_DATABASE_URI | 'sqlite:////tmp/job_store.db' | SQLAlchemy Database URL. See note about database backend below. |
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should change it from /tmp to /var/lib/ckan as @mbocevski suggested.

@abedkhooli
Copy link
Copy Markdown

Was this part of the 2.9.1 patch release? Can we move the /tmp/job_store.db to /var/lib/ckan in 2.9.1 or not yet (package install)?

@amercader
Copy link
Copy Markdown
Member Author

@abedkhooli datapusher has a separate release cycle than CKAN, so it's not related to the patch releases. Unless I'm missing something you can change the db location to /var/lib/ckan or something else right now as long as the folder is writable by the user running datapusher

@abedkhooli
Copy link
Copy Markdown

@amercader thanks. I moved it to /var/lib/ckan/jobstore/job_store.db but getting datapusher.jobs.HTTPError: <unprintable HTTPError object> after it fetches the csv file, determines field types and saving chunk 0 (last). The jobs table logs an error: internal server error which also shows in UI but no other clue what's going on.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Datapusher gets intermittently stuck when processing a large number of resources Replace sqlite with PostgreSQL Don't store job data in /tmp

4 participants