[CAI-821] Refresh vector index structured documents by mdciri · Pull Request #2101 · pagopa/developer-portal

mdciri · 2026-03-04T16:56:36Z

List of Changes

Add refresh vector index script for structured documents

Motivation and Context

In this way, there is no need to create from scratch the vector index using the create index method

How Has This Been Tested?

Screenshots (if appropriate):

Types of changes

Chore (nothing changes by a user perspective)
Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)

Checklist:

My change requires a change to the documentation.
I have updated the documentation accordingly.

* save query and evaluate functions refactoring * dynamodb query put * presidio * changeset * Update chatbot app queries * Update chatbot-monitor scripts * Update chatbot-monitor scripts * Update chatbot queries.py --------- Co-authored-by: mdciri <marco.cirillo@uqido.com> Co-authored-by: Marco Domenico Cirillo <59966344+mdciri@users.noreply.github.com>

* save query and evaluate functions refactoring * dynamodb query put * presidio * changeset * Update chatbot app queries * Update chatbot-evaluator modules * Update poetry files * Update lambda function * Update chatbot monitor scripts * Update chatbot-evaluate scripts * Update chatbot-evaluate lamda function * Update chatbot-monitor lambda function * Add changeset * Update chatbot-monitor monitor.py * Update chatbot-monitor monitor.py * Align to CAI-629 * Update lambda function to consider only unique records * Update lambda function in chatbot-monitor * Update chatbot-evaluate lambda function * Update chatbot-evaluate tests * Update loggers in chatbot monitor, index, and evaluate * Update loggers in chatbot, monitor, index, and evaluate * Update dockerfile for tests * Add new changeset * fix: logger param * fix: comment langfuse add score. We will use SQS queue in another pr * comment calls to presidio. In a future pr we will use SQS for them --------- Co-authored-by: batdevis <devis.battisti@uqido.com>

* Update chatbot settings * Update chatbot Dockerfiles * Update chatbot modules * Update chatbot-monitor to get release app version * Add changeset * Update version as tag product * Update changeset

* fix double entry * event test payload * docker compose test * pytest * pytest * docker compose test and volumes * aws motoserver file * move feedback update to chatbot-monitor * remove spacy download * fix test json file * changeset * remove duplicate test files * gitignore autoflake report files * remove unused imports * fix log level param * fix database envs for docker compose * chore: remove unused env var on .env.example * sqs local listener * docker compose * feat: clean workflow * docs * docker compose ports * chore: Dockerfiles refactor * Fix loop in the langfuse trace creation * Update spaces and indentation * remove logs * Update crate_langfuse_trace() method to respect API * Update crate_langfuse_trace() method to respect API * Removed recursive method in process_span() * Removed timestamp * Fix unique records to be processed * Removed timestamp to logger * fix bug in monitor.py * defence from NoneType values * Update poetry downgrading ragas * Evaluation accepts mock test * Update lambda function to wrtie mock scores * [CAI-729] ecoding decoding AWS SQS payload (#1984) * Update API to write in SQS monitor * Add compression and decompression scripts to chatbot, chatbot-evaluate and chatbot-monitor * Update scripts to compress and decompress payload from/to SQS * Add changeset * Update .changeset/pink-kings-enter.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * fix float number error on scores * fix compose * docker files cleanup * Update return type of process_span() method --------- Co-authored-by: mdciri <marco.cirillo@uqido.com> Co-authored-by: Marco Domenico Cirillo <59966344+mdciri@users.noreply.github.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Add package.json to parser app * Add changeset, update package-lock * Update apps/parser/package.json Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Add package.json to parser app * Add changeset, update package-lock * Update apps/parser/package.json Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Add puppeteer dependency in parser app * Add changeset * Update .changeset/three-cups-change.md Co-authored-by: marcobottaro <39835990+marcobottaro@users.noreply.github.com> * Update apps/parser/package.json Co-authored-by: marcobottaro <39835990+marcobottaro@users.noreply.github.com> --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: marcobottaro <39835990+marcobottaro@users.noreply.github.com>

…ndex

changeset-bot · 2026-03-04T16:57:08Z

🦋 Changeset detected

Latest commit: 6ff4e89

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package

Name	Type
chatbot-index	Minor

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

apps/chatbot-index/src/modules/url_handling.py

apps/chatbot-index/src/modules/vector_index.py

…nd functionality

…veloper-portal into CAI-821-update-cittadino-index

github-actions · 2026-03-05T17:24:32Z

Branch is not up to date with base branch

@mdciri it seems this Pull Request is not updated with base branch.
Please proceed with a merge or rebase to solve this.

apps/chatbot-index/src/modules/refresh_structured_docs.py

apps/chatbot-index/src/modules/vector_index.py

Co-authored-by: Noemi Manara <noemi.manara@gmail.com>

github-actions · 2026-03-12T08:38:59Z

Jira Pull Request Link

This Pull Request refers to the following Jira issue CAI-821

mdciri and others added 30 commits November 6, 2025 16:17

Update chatbot modules

baba529

Update chatbot queries.py

e52a688

Update poetry files

f38c31a

Update chatbot api queries

6b64198

Update chatbot dockerfile

e96dda1

Update chatbot-monitor scripts

763678a

Add changeset

295c6f7

Update chatbot notebook

afce9b6

Update chatbot src scripts

448eb3f

Merge branch 'main' into CAI-629-refactor-chatbot

97b3b16

Merge branch 'main' into CAI-629-refactor-chatbot

5caad3c

Merge branch 'main' into CAI-629-refactor-chatbot

7b0a9a3

Align to main

7e0e234

Merge branch 'main' into CAI-629-refactor-chatbot

66f7d2d

Merge branch 'main' into CAI-629-refactor-chatbot

cefe5ca

Merge alignment with main

f80554b

[CAI-684] Add langfuse release version (#1929)

6559fdd

* Update chatbot settings * Update chatbot Dockerfiles * Update chatbot modules * Update chatbot-monitor to get release app version * Add changeset * Update version as tag product * Update changeset

Merge branch 'main' into CAI-629-refactor-chatbot

dfdfa37

Merge branch 'main' into CAI-629-refactor-chatbot

e332f5c

Merge branch 'main' into CAI-629-refactor-chatbot

3589983

Refactor for multi index in redis

56cbc91

Merge branch 'main' into CAI-629-refactor-chatbot

6e57ae7

Copilot terraform skills

a0fe349

[CAI-747] add parser app (#1996)

24ad7a4

* Add package.json to parser app * Add changeset, update package-lock * Update apps/parser/package.json Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Update scripts to accept structured documents to create a dedicated i…

e779b68

…ndex

Update github action accordingly to python script

15b8b95

github-actions bot added the changeset label Mar 4, 2026

mdciri changed the title ~~[CAI-821] update cittadino indexj~~ [CAI-821] Refresh vector index structured documents Mar 4, 2026

github-actions bot added the size/small label Mar 4, 2026

github-actions bot assigned mdciri Mar 4, 2026

Merge branch 'main' into CAI-821-update-cittadino-index

19e1226

mdciri requested review from a team, MarBert, MarcoPonchia, marcobottaro and tommaso1 as code owners March 4, 2026 16:57

marcobottaro requested changes Mar 5, 2026

View reviewed changes

mdciri added 2 commits March 5, 2026 17:58

Refactor URL handling and vector index logging for improved clarity a…

cef2ee6

…nd functionality

Merge branch 'CAI-821-update-cittadino-index' of github.com:pagopa/de…

4097658

…veloper-portal into CAI-821-update-cittadino-index

anemone008 reviewed Mar 6, 2026

View reviewed changes

apps/chatbot-index/src/modules/refresh_structured_docs.py Show resolved Hide resolved

apps/chatbot-index/src/modules/vector_index.py Outdated Show resolved Hide resolved

mdciri mentioned this pull request Mar 11, 2026

[CAI-834] Add action to refresh structured documents #2116

Merged

6 tasks

MarBert and others added 3 commits March 11, 2026 11:22

Merge branch 'main' into CAI-821-update-cittadino-index

3f842f1

Update apps/chatbot-index/src/modules/refresh_structured_docs.py

86ef9cc

Co-authored-by: Noemi Manara <noemi.manara@gmail.com>

Update apps/chatbot-index/src/modules/vector_index.py

9e0fcf2

Co-authored-by: Noemi Manara <noemi.manara@gmail.com>

MarBert requested a review from marcobottaro March 11, 2026 10:31

MarBert and others added 4 commits March 11, 2026 12:58

Merge branch 'main' into CAI-821-update-cittadino-index

ec62d0d

Merge branch 'main' into CAI-821-update-cittadino-index

28d3324

Merge branch 'main' into CAI-821-update-cittadino-index

0bd10b5

Merge branch 'main' into CAI-821-update-cittadino-index

6ff4e89

MarBert approved these changes Mar 12, 2026

View reviewed changes

marcobottaro approved these changes Mar 12, 2026

View reviewed changes

marcobottaro merged commit 167d2ce into main Mar 12, 2026
12 checks passed

marcobottaro deleted the CAI-821-update-cittadino-index branch March 12, 2026 10:47

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[CAI-821] Refresh vector index structured documents#2101

[CAI-821] Refresh vector index structured documents#2101
marcobottaro merged 187 commits intomainfrom
CAI-821-update-cittadino-index

mdciri commented Mar 4, 2026 •

edited by marcobottaro

Loading

Uh oh!

changeset-bot bot commented Mar 4, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Mar 5, 2026

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Mar 12, 2026 •

edited by atlassian bot

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Conversation

mdciri commented Mar 4, 2026 • edited by marcobottaro Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

List of Changes

Motivation and Context

How Has This Been Tested?

Screenshots (if appropriate):

Types of changes

Checklist:

Uh oh!

changeset-bot bot commented Mar 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🦋 Changeset detected

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Mar 5, 2026

Branch is not up to date with base branch

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Mar 12, 2026 • edited by atlassian bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Jira Pull Request Link

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

mdciri commented Mar 4, 2026 •

edited by marcobottaro

Loading

changeset-bot bot commented Mar 4, 2026 •

edited

Loading

github-actions bot commented Mar 12, 2026 •

edited by atlassian bot

Loading