feat: enable drafting with knowledge by RolandMinrui · Pull Request #998 · microsoft/RD-Agent

RolandMinrui · 2025-06-27T09:13:52Z

Description

Motivation and Context

How Has This Been Tested?

If you are adding a new feature, test on your own test scripts.

Screenshots of Test Results (if appropriate):

Your own tests:

Types of changes

Fix bugs
Add new feature
Update documentation

📚 Documentation preview 📚: https://RDAgent--998.org.readthedocs.build/en/998/

…wledge_drafting

…Agent into knowledge_drafting

…wledge_drafting

you-n-g · 2025-07-01T09:16:11Z

rdagent/app/data_science/conf.py

    coding_fail_reanalyze_threshold: int = 3

-    debug_timeout: int = 600
+    debug_timeout: int = 900


Remember to change it back and change it with env.

you-n-g · 2025-07-01T09:17:37Z

rdagent/scenarios/data_science/dev/prompts.yaml

    - Step 1: If submission format has issues, prioritize fixing them before proceeding. If the format is correct and it's the first valid submission ever (there has never been valid submissions in the past), set `"Replace Best Result": "yes"`. If the format is correct and this is not the first valid submission, proceed to Step 2.
    - Step 2: If evaluation alignment issues are identified (validation approach does not follow competition requirements), address these methodological discrepancies immediately.
    - Step 3: If new results significantly worse than SOTA, or repeated hyperparameter adjustments yield no improvement, it might be time to rethink or shift focus.
+    - Step 4: If the result is only slightly better than the SOTA, but the code modifications are extensive (e.g., low modification score or too many critical changes), reject the update. Prefer small-step improvements with minimal changes. Set `"Replace Best Result": "no"` and explain in `"Reasoning"` starting with `[Code Change Too Large]`.


Is this PR focus on drafting?

If we want other changes can be evaluated alone, we can create a seperate PR.

you-n-g · 2025-07-01T10:00:25Z

rdagent/scenarios/data_science/proposal/exp_gen/draft.py



-class DSDraftExpGen(ExpGen):
+class CodingSketch(BaseModel):


We don't have to duplicate this. We can import it directly

you-n-g · 2025-07-01T10:08:08Z

rdagent/scenarios/data_science/proposal/exp_gen/proposal.py



+# TODO: merge the two version draft in the further
+def draft_exp_in_pipeline(scen: Scenario, trace: DSTrace) -> None | DSDraftExpGenV2:


We can use router to implement this feature

rdagent/scenarios/data_science/proposal/exp_gen/proposal.py

…Agent into knowledge_drafting

you-n-g · 2025-07-08T01:40:46Z

rdagent/core/conf.py

 )

-
+print('debug')


Clean the code.

you-n-g · 2025-07-08T01:41:31Z

rdagent/core/utils.py


 from filelock import FileLock
-from fuzzywuzzy import fuzz  # type: ignore[import-untyped]
+#from fuzzywuzzy import fuzz  # type: ignore[import-untyped]


Why remove this ?

you-n-g · 2025-07-08T01:41:42Z

rdagent/scenarios/data_science/dev/coder.py

@@ -0,0 +1 @@
+print('sss')


you-n-g · 2025-07-08T01:42:55Z

rdagent/scenarios/data_science/dev/draft_feedback/draft_v1.py

@@ -0,0 +1,17 @@
+
+from dev.feedback import DSExperiment2Feedback


We can have better folder structure.
Let's discussion this later

feedback/

init.py

draft.py

you-n-g · 2025-07-08T01:43:43Z

rdagent/scenarios/data_science/dev/feedback.py


        return hypothesis_feedback
+
+class DSDraftExperiment2Feedback(DSExperiment2Feedback):


Duplicated code

you-n-g · 2025-07-08T01:44:06Z

rdagent/scenarios/data_science/dev/prompts.yaml


  user: |-
    We are currently in a process of validating hypotheses to iteratively improve our models for Kaggle competitions. Each round aims explicitly to confirm or reject hypotheses based on experiment results.
+    We prioritize minimal, incremental code changes that lead to measurable improvements.**


Is this PR focus on drafting?

design a module.

Include + self customized.

you-n-g · 2025-07-08T01:44:18Z

rdagent/scenarios/data_science/loop.py

            self.trace = DSTrace(scen=scen)
-        self.summarizer = DSExperiment2Feedback(scen)
+
+        #self.summarizer = DSExperiment2Feedback(scen)


Suggested change

#self.summarizer = DSExperiment2Feedback(scen)

you-n-g · 2025-07-08T01:46:24Z

rdagent/scenarios/data_science/proposal/exp_gen/proposal.py

        # Drafting Stage
-        if draft_exp := draft_exp_in_decomposition(self.scen, trace):
-            return draft_exp
+        # if draft_exp := draft_exp_in_decomposition(self.scen, trace):


We can keep the orginal code and leave the TODO comments.

you-n-g · 2025-07-08T01:46:51Z

rdagent/scenarios/data_science/proposal/exp_gen/proposal.py

    ) -> DSExperiment:

        pipeline = DS_RD_SETTING.coder_on_whole_pipeline
-        if not pipeline and (draft_exp := draft_exp_in_decomposition(self.scen, trace)):


Do not break original logic if not necessary

you-n-g · 2025-07-08T01:47:26Z

rdagent/scenarios/data_science/proposal/exp_gen/router/__init__.py

+        return self.base_exp_gen.gen(trace)
+
+
+class DraftRouterExpGenV2(ExpGen):


Why do we have already two version of router alreadY?

…wledge_drafting

you-n-g · 2025-07-08T07:09:31Z

rdagent/scenarios/data_science/dev/draft_feedback/draft_v1.py

@@ -0,0 +1,17 @@
+
+from dev.feedback import DSExperiment2Feedback


feedback/

init.py

draft.py

you-n-g · 2025-07-08T07:20:45Z

rdagent/scenarios/data_science/dev/prompts.yaml


  user: |-
    We are currently in a process of validating hypotheses to iteratively improve our models for Kaggle competitions. Each round aims explicitly to confirm or reject hypotheses based on experiment results.
+    We prioritize minimal, incremental code changes that lead to measurable improvements.**


design a module.

Include + self customized.

rdagent/scenarios/data_science/proposal/exp_gen/draft.py

you-n-g · 2025-07-08T07:23:55Z

rdagent/scenarios/data_science/proposal/exp_gen/draft/prompts_draft.yaml

+    **[DataPreprocess] → [EDA] → [FeatureEngineer] → [Model] → [Tuning] → [Ensemble]**
+
+    ### Resource Note
+     **You have access to a 40GB V100 GPU.**  


you-n-g · 2025-07-08T07:24:38Z

rdagent/scenarios/data_science/proposal/exp_gen/draft/prompts_draft.yaml

+
+    ### Resource Note
+     **You have access to a 40GB V100 GPU.**  
+    You are **not restricted to lightweight models**. You may use medium to large pretrained architectures (e.g., EfficientNet-B3/B5, ConvNeXt-Tiny, Swin-S) **if they fit within the training time budget**.


Not so strict.

you-n-g · 2025-07-08T07:25:17Z

rdagent/scenarios/data_science/proposal/exp_gen/draft/prompts_draft.yaml

+
+    - During [Model] or [Tuning], always estimate and test the largest **batch size** that fits into available memory to **maximize throughput**.
+    - Use `nvidia-smi` or code-level profiling to find the optimal **batch size / num_workers**.
+    - If runtime is a constraint, assume a **budget of 1.5–2 hours total for training + inference** in the first version.


you-n-g · 2025-07-08T07:42:22Z

rdagent/scenarios/data_science/proposal/exp_gen/draft/prompts_draft.yaml

+
+
+
+# knowledge:


you-n-g · 2025-07-08T07:43:51Z

rdagent/scenarios/data_science/proposal/exp_gen/proposal.py

-        )
-    else:
-        return None
+# TODO: merge the two version draft in the further


remove this. It has been in router.

you-n-g · 2025-07-08T07:45:46Z

rdagent/scenarios/data_science/proposal/exp_gen/router/__init__.py

+        return self.base_exp_gen.gen(trace)
+
+
+"""


…wledge_drafting

you-n-g · 2025-07-08T08:54:47Z

rdagent/app/data_science/conf.py

    """Hypothesis generation class"""

+    summarizer: str = "rdagent.scenarios.data_science.dev.feedback.DSExperiment2Feedback"
+    summarizer_version: str = "exp_feedback"  # exp_feedback or exp_feedback_draft


summarizer_init_kwargs: dict = {.....}

you-n-g · 2025-07-08T08:55:13Z

rdagent/scenarios/data_science/loop.py

            self.trace = DSTrace(scen=scen)
-        self.summarizer = DSExperiment2Feedback(scen)
+
+        self.summarizer = import_class(PROP_SETTING.summarizer)(scen=scen, version=PROP_SETTING.summarizer_version)


self.summarizer = import_class(PROP_SETTING.summarizer)(scen=scen, **)

you-n-g · 2025-07-08T08:55:44Z

rdagent/scenarios/data_science/proposal/exp_gen/draft/draft.py

+from rdagent.utils.agent.tpl import T
+
+
+class CodingSketch(BaseModel):


import from utils

after merge master

…wledge_drafting

* add pipeline for drafting v2 * fix the pipeline and add general knowledge * debug * fix bug * fix bug * change draft version1 * add function calling to task gen * fix circular import bug * change draft version3 * exp1_test * feat: add DraftRouterExpGen and make summarizer configurable * Update rdagent/scenarios/data_science/proposal/exp_gen/proposal.py * change code structure * stashed changes * test * test1 * revert conf.py * add runtime enviornment info to general knowledge * remove redundant code * clean code * remove files * reformat * fix bug * fix bug * simplify code * fix minor bug * fix bug and reformat * revert config * remove unused prompt * add general knowledge * fix ci --------- Co-authored-by: Xu <v-xuminrui@microsoft.com> Co-authored-by: jingyuanlm <842442862@qq.com> Co-authored-by: Young <afe.young@gmail.com> Co-authored-by: you-n-g <you-n-g@users.noreply.github.com>

Xu added 3 commits June 26, 2025 09:50

add pipeline for drafting v2

960db1f

fix the pipeline and add general knowledge

6a38e4f

Merge branch 'main' of https://github.com/microsoft/RD-Agent into kno…

c0dc364

…wledge_drafting

RolandMinrui marked this pull request as draft June 27, 2025 09:14

jingyuanlm and others added 11 commits June 27, 2025 10:18

debug

454abb0

fix bug

91f374c

fix bug

ae1fbed

change draft version1

b4c18b2

Merge branch 'main' of https://github.com/microsoft/RD-Agent into kno…

a566b6a

…wledge_drafting

add function calling to task gen

dfa0fcf

fix circular import bug

c4ab4b7

change draft version3

3abba61

Merge branch 'knowledge_drafting' of https://github.com/microsoft/RD-…

dd4f276

…Agent into knowledge_drafting

Merge branch 'main' of https://github.com/microsoft/RD-Agent into kno…

daacb28

…wledge_drafting

exp1_test

6aee573

you-n-g reviewed Jul 4, 2025

View reviewed changes

feat: add DraftRouterExpGen and make summarizer configurable

9b73299

you-n-g reviewed Jul 4, 2025

View reviewed changes

rdagent/scenarios/data_science/proposal/exp_gen/proposal.py Show resolved Hide resolved

you-n-g and others added 4 commits July 4, 2025 16:17

Update rdagent/scenarios/data_science/proposal/exp_gen/proposal.py

1456e0f

change code structure

f632df6

Merge branch 'knowledge_drafting' of https://github.com/microsoft/RD-…

a33765a

…Agent into knowledge_drafting

stashed changes

9329487

jingyuanlm force-pushed the knowledge_drafting branch from 747dee1 to 9329487 Compare July 7, 2025 08:57

jingyuanlm added 2 commits July 7, 2025 09:00

test

ecf3ec4

test1

b6155b3

you-n-g reviewed Jul 8, 2025

View reviewed changes

Xu added 4 commits July 8, 2025 07:21

Merge branch 'main' of https://github.com/microsoft/RD-Agent into kno…

ca79889

…wledge_drafting

revert conf.py

29cc6ea

Merge branch 'main' of https://github.com/microsoft/RD-Agent into kno…

15a42bb

…wledge_drafting

add runtime enviornment info to general knowledge

bf43240

you-n-g reviewed Jul 8, 2025

View reviewed changes

Xu added 7 commits July 8, 2025 07:47

remove redundant code

8a74a8d

clean code

a148bee

remove files

137c7b7

reformat

e3884b8

Merge branch 'main' of https://github.com/microsoft/RD-Agent into kno…

64708ba

…wledge_drafting

fix bug

962c3a4

fix bug

ef0c985

you-n-g reviewed Jul 8, 2025

View reviewed changes

Xu added 8 commits July 8, 2025 09:02

simplify code

d64514f

fix minor bug

bca4c5d

fix bug and reformat

78ec125

revert config

1953c88

Merge branch 'main' of https://github.com/microsoft/RD-Agent into kno…

d49ccea

…wledge_drafting

remove unused prompt

f3435a0

add general knowledge

c28f861

fix ci

83d975e

RolandMinrui marked this pull request as ready for review July 9, 2025 08:30

RolandMinrui merged commit 8e385eb into main Jul 9, 2025
9 checks passed

RolandMinrui deleted the knowledge_drafting branch July 9, 2025 10:36

you-n-g mentioned this pull request Jul 9, 2025

chore(main): release 0.8.0 #1030

Merged



		# TODO: merge the two version draft in the further
		def draft_exp_in_pipeline(scen: Scenario, trace: DSTrace) -> None \| DSDraftExpGenV2:

		@@ -0,0 +1,17 @@

		from dev.feedback import DSExperiment2Feedback


		return hypothesis_feedback

		class DSDraftExperiment2Feedback(DSExperiment2Feedback):

		return self.base_exp_gen.gen(trace)


		class DraftRouterExpGenV2(ExpGen):

		from rdagent.utils.agent.tpl import T


		class CodingSketch(BaseModel):

		)


		print('debug')

		@@ -0,0 +1 @@
		print('sss') No newline at end of file

Uh oh!

Conversation

RolandMinrui commented Jun 27, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Motivation and Context

How Has This Been Tested?

Screenshots of Test Results (if appropriate):

Types of changes

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

RolandMinrui commented Jun 27, 2025 •

edited by github-actions bot

Loading