API BehaviorTree

API: BehaviorTree

BehaviorTree is a tiny functional behavior-tree helper used by RBXStateMachine for decision composition. Nodes are plain functions that take a context object and return one of three status strings.

Important

This page documents FSM/Orchestrator/Core/Factory/BehaviorTree.luau and example usage patterns from RetryBehavior.luau and ServiceBrain.luau.

---

Overview

A behavior tree node in this framework is simply:

(context: any) -> "Success" | "Failure" | "Running"

The module intentionally stays small:

• Selector
• Sequence
• Inverter
• Succeeder
• Condition
• SetState
• BehaviorTreeStatus

Status enum

BehaviorTree.BehaviorTreeStatus = {
    Success = "Success",
    Failure = "Failure",
    Running = "Running",
}

Status	Meaning
Success	node finished successfully
Failure	node failed and parent should continue / stop according to its own semantics
Running	node is still in progress; composites pass it upward without converting it

---

Execution Model

Selector vs Sequence

Selector
┌──────────┐
│ Selector │
└────┬─────┘
     ▼
┌─────────────┐
│ Run child 1 │
└────┬────────┘
     ▼
┌──────────┐
│ Failure? │
└─┬────┬───┘
  │yes │no
  ▼    ▼
┌──────────────┐   ┌─────────────────────┐
│ Run next child │  │ Return child status │
└──────┬────────┘   └─────────────────────┘
       ▼
┌──────────────────────────────────────────┐
│ Repeat until non-Failure or no children │
└──────────────────────────────────────────┘

Sequence
┌──────────┐
│ Sequence │
└────┬─────┘
     ▼
┌─────────────┐
│ Run child 1 │
└────┬────────┘
     ▼
┌──────────┐
│ Success? │
└─┬────┬───┘
  │yes │no
  ▼    ▼
┌──────────────┐   ┌─────────────────────┐
│ Run next child │  │ Return child status │
└──────┬────────┘   └─────────────────────┘
       ▼
┌────────────────────────────────────────┐
│ Repeat until non-Success or all pass │
└────────────────────────────────────────┘

Invalid return normalization

Every composite / decorator uses an internal normalization step. If a child returns anything other than the three official status strings, it is coerced to Failure.

That includes:

• nil
• unexpected strings
• accidental booleans
• any other unsupported return type

This makes the tree fail safe instead of crashing the whole caller.

---

Initialization

BehaviorTree.Initialize(Util)

BehaviorTree.Initialize(Util: table) -> ()

One-time setup called by Factory.

• @reads: Initialized
• @writes: module logger, initialized flag

If you try to construct nodes before initialization, the module asserts.

---

Core Constructors

BehaviorTree.Selector(children)

BehaviorTree.Selector(children: { (context: any) -> string }) -> (context: any) -> string

Runs children in order and returns the first child that is not Failure.

Returns

• Success if a child succeeds
• Running if a child reports running before any success
• Failure only if every child fails or normalizes to failure

Side effects

• @reads: child return values
• @writes: none directly

Example

local BT = shared.fsm.BehaviorTree

local chooseAction = BT.Selector({
    BT.Sequence({
        BT.Condition(function(ctx) return ctx.Health < 20 end),
        BT.SetState("Flee"),
    }),
    BT.Sequence({
        BT.Condition(function(ctx) return ctx.Target ~= nil end),
        BT.SetState("Attack"),
    }),
    BT.SetState("Patrol"),
})

BehaviorTree.Sequence(children)

BehaviorTree.Sequence(children: { (context: any) -> string }) -> (context: any) -> string

Runs children in order and returns the first child that is not Success.

Returns

• Failure if any child fails
• Running if any child is still running before a failure occurs
• Success only if every child succeeds

Example

local attackSequence = BT.Sequence({
    BT.Condition(function(ctx)
        return ctx.Target ~= nil and ctx.TargetDistance < 10
    end),
    function(ctx)
        ctx.FSM:ChangeState({ Name = "Attack" })
        return BT.BehaviorTreeStatus.Success
    end,
})

BehaviorTree.Inverter(child)

BehaviorTree.Inverter(child: (context: any) -> string) -> (context: any) -> string

Flips Success and Failure, while passing Running through unchanged.

Example

local noTarget = BT.Inverter(BT.Condition(function(ctx)
    return ctx.Target ~= nil
end))

BehaviorTree.Succeeder(child)

BehaviorTree.Succeeder(child: (context: any) -> string) -> (context: any) -> string

Converts any final status to Success, but preserves Running.

This is useful when a side-effecting node is optional and should not break the parent sequence.

Example

local safeTelemetry = BT.Succeeder(function(ctx)
    local ok, err = pcall(ctx.PushMetrics)
    return ok and BT.BehaviorTreeStatus.Success or BT.BehaviorTreeStatus.Failure
end)

BehaviorTree.Condition(predicate)

BehaviorTree.Condition(predicate: (context: any) -> boolean) -> (context: any) -> string

Wraps a boolean predicate.

Returns

• Success when predicate(context) is truthy
• Failure otherwise

Example

local hasTarget = BT.Condition(function(ctx)
    return ctx.Target ~= nil
end)

BehaviorTree.SetState(stateName)

BehaviorTree.SetState(stateName: string) -> (fsm: any) -> string

Returns a leaf node that directly assigns fsm.State = stateName if the state differs, then returns Success.

Side effects

• @reads: fsm.State
• @writes: fsm.State

Example

local goIdle = BT.SetState("Idle")

Caution

SetState writes the State property directly. In RBXStateMachine, that means the real validated transition happens on the next FSM _Update() pass. If you need immediate validation, OnLeave, history, and signal ordering right now, use a custom leaf that calls fsm:ChangeState(...) instead.

---

Composition Patterns

Pattern 1: pure FSM steering with SetState

local brain = BT.Selector({
    BT.Sequence({
        BT.Condition(function(fsm) return fsm.Context.Health < 15 end),
        BT.SetState("Flee"),
    }),
    BT.Sequence({
        BT.Condition(function(fsm) return fsm.Context.Target ~= nil end),
        BT.SetState("Attack"),
    }),
    BT.SetState("Patrol"),
})

self:AddState("Think", {
    OnHeartbeat = function(_, fsm)
        brain(fsm)
    end,
}, { "Flee", "Attack", "Patrol" })

Pattern 2: explicit transition leaf using ChangeState

Use this when you need the transition to happen immediately inside the node.

local BT = shared.fsm.BehaviorTree

local function ChangeStateNode(targetState)
    return function(fsm)
        fsm:ChangeState({ Name = targetState })
        return BT.BehaviorTreeStatus.Success
    end
end

local doorDecisionTree = BT.Selector({
    BT.Sequence({
        BT.Condition(function(fsm)
            return fsm.Context.IsLocked == true
        end),
        ChangeStateNode("Locked"),
    }),
    BT.Sequence({
        BT.Condition(function(fsm)
            return fsm.Context.ShouldOpen == true
        end),
        ChangeStateNode("Opening"),
    }),
    ChangeStateNode("Idle"),
})

Pattern 3: using Running

Running short-circuits the parent without converting to success/failure. That lets you model multi-tick work.

local waitForAnimation = function(ctx)
    if not ctx.AnimationTrack then
        return BT.BehaviorTreeStatus.Failure
    end

    if ctx.AnimationTrack.IsPlaying then
        return BT.BehaviorTreeStatus.Running
    end

    return BT.BehaviorTreeStatus.Success
end

---

Real Framework Examples

RetryBehavior example (actual framework pattern)

FSM/Orchestrator/Core/DataStore/RetryBehavior.luau uses a selector with ordered fallback:

local Tree = BT.Selector({
    BT.Sequence({
        AttemptCall,
        CompleteRequest,
    }),
    BT.Sequence({
        CanRetry,
        ScheduleRetry,
    }),
    FailRequest,
})

This means:

1. try the request
2. if it succeeds, mark the FSM successful
3. if it fails, check whether retry budget remains
4. if retries remain, increment RetryCount, set FSM.WaitSpan, and move to Retrying
5. otherwise fail permanently

ServiceManager “Nexus brain” style example

The ServiceManager root behavior (ServiceBrain.Build) uses a large Sequence / Selector composition to gate initialization, refresh data only when needed, and always tick animations.

local tree = BehaviorTree.Sequence({
    BehaviorTree.Condition(function(ctx)
        return ctx.Nexus and ctx.Nexus._initialized
    end),

    SyncContextData,

    BehaviorTree.Selector({
        BehaviorTree.Sequence({
            NeedsDataRefresh,
            FetchSubsystemData,
            BehaviorTree.Succeeder(RunInsightAnalysis),
            RenderActiveSubsystem,
        }),
        BehaviorTree.Sequence({
            HasDirtyUI,
            RenderActiveSubsystem,
        }),
        function()
            return BehaviorTree.BehaviorTreeStatus.Success
        end,
    }),

    BehaviorTree.Succeeder(UpdateAnimations),
})

That pattern is useful whenever you want:

• a hard precondition gate at the top
• a best-effort analysis step that should not fail the whole tree
• a fallback branch that still reports success when no expensive work is necessary

---

Integration with BaseStateMachine

Recommended integration points

Place	When to use it
OnHeartbeat	decision tree reevaluated every FSM update
OnEnter	one-shot branch selection when entering a state
dedicated function state	tree becomes the entire behavior of the state

Example: behavior tree inside an FSM state object

function EnemyFSM:RegisterStates()
    local BT = shared.fsm.BehaviorTree

    local decide = BT.Selector({
        BT.Sequence({
            BT.Condition(function(fsm)
                return fsm.Context.Health and fsm.Context.Health < 20
            end),
            BT.SetState("Retreat"),
        }),
        BT.Sequence({
            BT.Condition(function(fsm)
                return fsm.Context.Target ~= nil
            end),
            BT.SetState("Attack"),
        }),
        BT.SetState("Patrol"),
    })

    self:AddState("Think", {
        OnHeartbeat = function(_, fsm)
            decide(fsm)
        end,
    }, { "Retreat", "Attack", "Patrol" })
end

Example: custom context instead of the FSM itself

local tree = BT.Sequence({
    BT.Condition(function(ctx) return ctx.Entity ~= nil end),
    function(ctx)
        ctx.Entity.Health = math.max(0, (ctx.Entity.Health or 0) - 5)
        return BT.BehaviorTreeStatus.Success
    end,
})

tree({ Entity = entity })

---

Edge Cases & Gotchas

1. Invalid child returns become Failure. If you accidentally forget return Status.Success, the composite treats that child as failure.
2. SetState is deferred in practice. It only writes fsm.State; BaseStateMachine later notices the change and performs the real transition.
3. Running is preserved by decorators. Inverter and Succeeder do not convert it.
4. There is no built-in memory / blackboard system. Your context object is the blackboard.
5. Selectors stop on Running. Because Running ~= Failure, a selector will not evaluate later children once a child reports running.
6. Sequences stop on Running. Because Running ~= Success, a sequence also short-circuits until the next tick.
7. Nodes are plain functions. Any mutation, yielding, logging, or FSM calls come entirely from your leaf logic.
8. Initialization is mandatory. Factory handles this for you; standalone manual use must still call BehaviorTree.Initialize(...) first.

---

Related Pages

• API: BaseStateMachine
• API: Factory
• API: Orchestrator
• Best Practices
• Examples