Skip to content

feat(auto_source): add support for auto_source feature#676

Merged
gildesmarais merged 25 commits into
masterfrom
feat/auto-source-integration
Oct 21, 2024
Merged

feat(auto_source): add support for auto_source feature#676
gildesmarais merged 25 commits into
masterfrom
feat/auto-source-integration

Conversation

@gildesmarais
Copy link
Copy Markdown
Member

@gildesmarais gildesmarais commented Aug 17, 2024 •
edited
Loading

Depends on html2rss/html2rss#202 || v0.14.0

  1. setup of roda 'complex app' file structure utilizing hash_branches with hash_branch_view_subdir
  2. added previously missing specs for app
  3. added autosource integration
    a. made it configurable via env variables
    b. setup rspec to use code climate to test using env variables
    c. setup rspec to issue http requests to test http endpoints
    d. configured vcr hook into ssrf-filter issued http requests (in parallel with faraday)
    e. Build for a single user which requires request to originate from allowed hostnames
    f. auto sourced feeds use the global stylesheet
    g. added html/js/css to provide gui with some use of use tools, i.e. open in feed reader and setup roda to use content_for hooks
Details

This pull request introduces several enhancements and new features, focusing on dependency updates, environment configuration, and the addition of an automatic feed generation feature. The most important changes include updating dependencies in the Gemfile, adding new environment variables, and implementing the auto source feature with its corresponding documentation and helper methods.

Dependency Updates:

  • Updated html2rss gem to version ~> 0.14 and added base64 and ssrf_filter gems to the Gemfile.
  • Added climate_control, rack-test, and webmock gems to the test group in the Gemfile.

Environment Configuration:

  • Modified environment variables in docker-compose.yml and provided documentation on how to enable and configure the auto source feature in the README.md. [1] [2] [3]

Auto Source Feature:

  • Added helpers/auto_source.rb to handle the auto source feature, including methods for enabling the feature, managing credentials, and checking request origins.
  • Updated public/auto_source.css to style the auto source form and its components.

Codebase Improvements:

  • Simplified development? method and adjusted CSP settings in app.rb. [1] [2]
  • Refactored error handling by renaming helpers/error_handlers.rb to helpers/handle_error.rb and adding a new error case for Html2rss::Error. [1] [2]

Other Changes:

  • Moved Rack::Timeout middleware usage to non-development environments in config.ru. [1] [2]
  • Updated content type to CONTENT_TYPE_RSS in helper methods for handling HTML2RSS configs. [1] [2]

@gildesmarais gildesmarais changed the title development env checks feat: enable use of auto_source Aug 17, 2024
@gildesmarais gildesmarais force-pushed the feat/auto-source-integration branch 3 times, most recently from 13c5037 to 920463e Compare August 20, 2024 14:25
@gildesmarais gildesmarais force-pushed the feat/auto-source-integration branch 2 times, most recently from 86707f1 to d04dfb3 Compare October 8, 2024 09:59
@gildesmarais gildesmarais mentioned this pull request Oct 8, 2024
Merged
@gildesmarais gildesmarais changed the title feat: enable use of auto_source feat(auto_source): add support for auto_source feature Oct 8, 2024
@gildesmarais gildesmarais force-pushed the feat/auto-source-integration branch from d04dfb3 to 8714ceb Compare October 8, 2024 12:21
@gildesmarais gildesmarais force-pushed the feat/auto-source-integration branch 3 times, most recently from 2c78fd3 to 48cc618 Compare October 20, 2024 12:37
@gildesmarais
feat: add development? env check
6ea0014
@gildesmarais
feat: expose endpoint which calls H2r.auto_source
1ffeb50
@gildesmarais
feat: use ssrf_filter to request user provided urls
24734c7
@gildesmarais
feat: use hash_branch_view_dir
0070399
@gildesmarais
feat: adapt to changed auto_source constructor signature
7588413
@gildesmarais
feat: add auto_source#index
b2da912
@gildesmarais
feat: use content-type constant
1d6fd45
@gildesmarais
feat: rack-timeout only when RACK_ENV!=development
8d98dff
@gildesmarais
fix: rendering of iframe prevented
b2fedc8 
Signed-off-by: Gil Desmarais <git@desmarais.de>
@gildesmarais
refactor: improve auto_source form handling and styles
97a8556
@gildesmarais
feat(auto_source): automatically submit form when ?url= is present
f577f1b
@gildesmarais
feat(auto_source): integrate content_for plugin and enhance UX
621a154 
Added `content_for` plugin to manage CSS and scripts. Implemented new bookmarklet, button handlers, and form improvements for better user interaction.

Signed-off-by: Gil Desmarais <git@desmarais.de>
@gildesmarais
fix(auto_source): update index.erb to improve input validation and UX
d9a076d 
Enhanced URL pattern to exclude user info, added autofocus for better user experience.
@gildesmarais
feat: assert request sources from allowed host/origin
98a3ea0
@gildesmarais gildesmarais force-pushed the feat/auto-source-integration branch 4 times, most recently from e6b7016 to 4d47f02 Compare October 20, 2024 16:41
@gildesmarais gildesmarais force-pushed the feat/auto-source-integration branch 4 times, most recently from c8b8d4e to d74902b Compare October 21, 2024 07:10
@gildesmarais gildesmarais force-pushed the feat/auto-source-integration branch from d74902b to 397db79 Compare October 21, 2024 07:21
@gildesmarais gildesmarais marked this pull request as ready for review October 21, 2024 07:24
@gildesmarais gildesmarais merged commit 531dced into master Oct 21, 2024
@gildesmarais gildesmarais deleted the feat/auto-source-integration branch October 21, 2024 07:26
@github-actions github-actions Bot mentioned this pull request May 1, 2026
Closed
This was referenced May 1, 2026
Closed
Merged
gildesmarais added a commit that referenced this pull request May 1, 2026
@gildesmarais @github-actions
chore(main): release html2rss-web 1.1.0 (#971)
1da518f 
🤖 I have created a release *beep* *boop*
---


##
[1.1.0](html2rss-web-v1.0.0...html2rss-web/v1.1.0)
(2026-05-01)


### Features

* add help text on error page
([eeee345](eeee345)),
closes [#338](#338)
* add routed frontend feed creation workflow
([#963](#963))
([2d1b71a](2d1b71a))
* **auto_source:** add support for `auto_source` feature
([#676](#676))
([531dced](531dced))
* default browserless onboarding and request strategies
([#895](#895))
([377cff0](377cff0))
* **deps:** use html2rss in latest development status
([#728](#728))
([5885d1d](5885d1d))
* **docker:** switch to alpine 21
([7adcc89](7adcc89))
* **docker:** upgrade to use ruby 3.3 image
([ceafe24](ceafe24))
* **docker:** use multilayer build to cut image size in half
([2f6e322](2f6e322))
* **docker:** use Ruby 3.4
([4f7d795](4f7d795))
* **frontend:** polish result experience and validation tooling
([#964](#964))
([b11665e](b11665e))
* **frontend:** relaunch the app with a focused v1 flow
([e0692d7](e0692d7))
* **frontend:** unify feed/result state flow
([#943](#943))
([6dfa1a9](6dfa1a9))
* **health_check:** add HTTP Basic authentication to `GET
/health_check.txt`
([#559](#559))
([d0ccd83](d0ccd83))
* improve example feed config in feed.yml and link to it
([#552](#552))
([de08695](de08695))
* install Gemfile.lock specified bundler version
([4190160](4190160))
* integrate request_service and use ssrf_filter strategy by default
([#707](#707))
([b7516fd](b7516fd))
* link included feeds to the instance feed directory
([#901](#901))
([51ce79a](51ce79a))
* optionally allow APM using Sentry via env variable
([#696](#696))
([94477d5](94477d5))
* redact sensitive feed data in structured logs
([#903](#903))
([ee7df73](ee7df73))
* remove dependency on activesupport
([048cb73](048cb73))
* **runtime:** rebuild feed and api behavior around typed v1 services
([b61602d](b61602d))
* simplify feed creation contract & backend error handling
([#962](#962))
([dfca027](dfca027))
* stabilize public http interface & slimmer docker
([#882](#882))
([fe3f4be](fe3f4be))
* unify web and feed result surfaces
([#896](#896))
([e747b23](e747b23))
* use parallel processing for feed retrieval in health_check.rb
([#665](#665))
([4a24997](4a24997))


### Bug Fixes

* ArgumentError when RACK_TIMEOUT_SERVICE_TIMEOUT env var is set
([96acbab](96acbab)),
closes [#527](#527)
* **auto_source:** respect headers from global config
([#691](#691))
([3e9ba91](3e9ba91))
* **build:** only cleanup when there is a test container
([f7bafa6](f7bafa6))
* caching with dynamic parameters yields incorrect rss
([#589](#589))
([bb945c2](bb945c2)),
closes [#587](#587)
* **ci:** repair Ruby, OpenAPI, and frontend checks
([#880](#880))
([ec6673b](ec6673b))
* defects for token/retry/loading UX
([#924](#924))
([2d38633](2d38633))
* **docker:** missing curl installation for health check
([0bd9157](0bd9157))
* example feed in config/feeds.yml broken
([#664](#664))
([b961897](b961897))
* **frontend:** preserve created feeds when preview loading fails
([#915](#915))
([383ecc3](383ecc3))
* **frontend:** streamline web ux
([#916](#916))
([85e79bf](85e79bf))
* harden container config defaults
([392997c](392997c))
* healthcheck broken due to missing curl
([c97e746](c97e746))
* keep unknown api v1 paths inside the api contract
([a820478](a820478))
* responds with http status 422
([#738](#738))
([ad9394c](ad9394c))
* **runtime:** polish relaunch smoke behavior and health checks
([65e1644](65e1644))
* stylesheets not included in feed
([#779](#779))
([9116d9d](9116d9d))
* tzdata package not installed but required for tz conversion
([#663](#663))
([55814d2](55814d2))
* **web:** harden feed reader fallback and rss rendering
([#944](#944))
([438d9f6](438d9f6))
* **web:** harden observability env handling and Sentry log redaction
([#917](#917))
([ed2b3e9](ed2b3e9))


### Performance Improvements

* enable YJIT
([729f31f](729f31f))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).

---------

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
@gildesmarais gildesmarais mentioned this pull request May 1, 2026
Merged
gildesmarais added a commit that referenced this pull request May 1, 2026
@gildesmarais @github-actions
chore(main): release 1.2.0 (#973)
eb01ec4 
🤖 I have created a release *beep* *boop*
---


##
[1.2.0](v1.1.0...v1.2.0)
(2026-05-01)


### Features

* add help text on error page
([eeee345](eeee345)),
closes [#338](#338)
* add routed frontend feed creation workflow
([#963](#963))
([2d1b71a](2d1b71a))
* **auto_source:** add support for `auto_source` feature
([#676](#676))
([531dced](531dced))
* default browserless onboarding and request strategies
([#895](#895))
([377cff0](377cff0))
* **deps:** use html2rss in latest development status
([#728](#728))
([5885d1d](5885d1d))
* **docker:** switch to alpine 21
([7adcc89](7adcc89))
* **docker:** upgrade to use ruby 3.3 image
([ceafe24](ceafe24))
* **docker:** use multilayer build to cut image size in half
([2f6e322](2f6e322))
* **docker:** use Ruby 3.4
([4f7d795](4f7d795))
* **frontend:** polish result experience and validation tooling
([#964](#964))
([b11665e](b11665e))
* **frontend:** relaunch the app with a focused v1 flow
([e0692d7](e0692d7))
* **frontend:** unify feed/result state flow
([#943](#943))
([6dfa1a9](6dfa1a9))
* **health_check:** add HTTP Basic authentication to `GET
/health_check.txt`
([#559](#559))
([d0ccd83](d0ccd83))
* improve example feed config in feed.yml and link to it
([#552](#552))
([de08695](de08695))
* install Gemfile.lock specified bundler version
([4190160](4190160))
* integrate request_service and use ssrf_filter strategy by default
([#707](#707))
([b7516fd](b7516fd))
* link included feeds to the instance feed directory
([#901](#901))
([51ce79a](51ce79a))
* optionally allow APM using Sentry via env variable
([#696](#696))
([94477d5](94477d5))
* redact sensitive feed data in structured logs
([#903](#903))
([ee7df73](ee7df73))
* remove dependency on activesupport
([048cb73](048cb73))
* **runtime:** rebuild feed and api behavior around typed v1 services
([b61602d](b61602d))
* simplify feed creation contract & backend error handling
([#962](#962))
([dfca027](dfca027))
* stabilize public http interface & slimmer docker
([#882](#882))
([fe3f4be](fe3f4be))
* unify web and feed result surfaces
([#896](#896))
([e747b23](e747b23))
* use parallel processing for feed retrieval in health_check.rb
([#665](#665))
([4a24997](4a24997))


### Bug Fixes

* ArgumentError when RACK_TIMEOUT_SERVICE_TIMEOUT env var is set
([96acbab](96acbab)),
closes [#527](#527)
* **auto_source:** respect headers from global config
([#691](#691))
([3e9ba91](3e9ba91))
* **build:** only cleanup when there is a test container
([f7bafa6](f7bafa6))
* caching with dynamic parameters yields incorrect rss
([#589](#589))
([bb945c2](bb945c2)),
closes [#587](#587)
* **ci:** repair Ruby, OpenAPI, and frontend checks
([#880](#880))
([ec6673b](ec6673b))
* **ci:** robustly parse release tags and align config
([#972](#972))
([2efd6ef](2efd6ef))
* defects for token/retry/loading UX
([#924](#924))
([2d38633](2d38633))
* **docker:** missing curl installation for health check
([0bd9157](0bd9157))
* example feed in config/feeds.yml broken
([#664](#664))
([b961897](b961897))
* **frontend:** preserve created feeds when preview loading fails
([#915](#915))
([383ecc3](383ecc3))
* **frontend:** streamline web ux
([#916](#916))
([85e79bf](85e79bf))
* harden container config defaults
([392997c](392997c))
* healthcheck broken due to missing curl
([c97e746](c97e746))
* keep unknown api v1 paths inside the api contract
([a820478](a820478))
* responds with http status 422
([#738](#738))
([ad9394c](ad9394c))
* **runtime:** polish relaunch smoke behavior and health checks
([65e1644](65e1644))
* stylesheets not included in feed
([#779](#779))
([9116d9d](9116d9d))
* tzdata package not installed but required for tz conversion
([#663](#663))
([55814d2](55814d2))
* **web:** harden feed reader fallback and rss rendering
([#944](#944))
([438d9f6](438d9f6))
* **web:** harden observability env handling and Sentry log redaction
([#917](#917))
([ed2b3e9](ed2b3e9))


### Performance Improvements

* enable YJIT
([729f31f](729f31f))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).

---------

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

Project Board
Status: Done

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@gildesmarais