RustScan Scripting Engine

[bee-san]

TL;DR

• Write exploit scripts in Python. Users can do whatever they want. Get robots.txt, check if FTP is anonymous.
• Use a custom tagging system (set theory based and argument based) to allow the user to create their own perfect categories of scripts.

Problem

Users want to write exploit modules with RustScan. Specifically, they feel ike Nmap's LUA isn't modern enough, or they want to pipe the output of RustScan into the input of a script.

They want to write exploit scripts using RustScan.

But we, the core maintainers of RustScan, cannot possibly compete with Nmap for scripts -- especially when we are focussed mostly on improving RustScan.

Solution

We introduce a RustScan Scripting Engine which will allow anyone to create scripts, or custom exploits and use them in RustScan.

This way, the community can build scripts and exploits if they want -- while we are RustScan work on improving the scanner generally.

Python

We should use Python to allow users to write scripts. Rust. Firstly, Rust has always had very good support for foreign function interfaces (FFI) with Python. You can even write inline Python with Rust.

We should consider using Pyo3.
https://github.com/PyO3/pyo3.

Python is easily the most friendly programming language around, Many people know it, and a lot of exploits (from Exploit-DB) are written in Python.

Giving the user the ability to not only write custom scripts for RustScan, but to do so in Python would be a huge plus.

Default template

We need to create a default template, so RustScan can easily intrepret the template and use it.

The exact template is unknown, until we look at Pyo3. However, Pyo3 has this Rust code (for calling Python from Rust):

use pyo3::prelude::*;
use pyo3::types::IntoPyDict;

fn main() -> Result<(), ()> {
    Python::with_gil(|py| {
        main_(py).map_err(|e| {
          // We can't display Python exceptions via std::fmt::Display,
          // so print the error here manually.
          e.print_and_set_sys_last_vars(py);
        })
    })
}

fn main_(py: Python) -> PyResult<()> {
    let sys = py.import("sys")?;
    let version: String = sys.get("version")?.extract()?;
    let locals = [("os", py.import("os")?)].into_py_dict(py);
    let code = "os.getenv('USER') or os.getenv('USERNAME') or 'Unknown'";
    let user: String = py.eval(code, None, Some(&locals))?.extract()?;
    println!("Hello {}, I'm Python {}", user, version);
    Ok(())
}

We should have an array (or vector) at the top of the script as an attribute. This vector will describe the tags the script has. Which is talked about next.

We should also either fetch the documentation from the documentation comments /// if possible, or suggest the user fill in attributes for documentation.

Preferably, the user will write something like:

tags = [core_approved, http, robotstxt, hyper, get_request]
ports = [80]
name = "Robots.txt grabbing"
description = "Grabs the robots.txt file from webservers"

Ideally, we would have either another script which can detect when a port is a web-server, which the user can import and use in their Python script.

Note in the Pyo3 script, it is possible to import other Python modules and use them.

However, also note that Pyo3 looks a bit alien. Check out the documentation to see how alien it looks:

https://pyo3.rs/v0.11.1/python_from_rust.html.

We may need to choose another FFI library, we may not. Please discuss in the comments.

Perhaps Rust-cpython:
https://github.com/dgrunwald/rust-cpython.

For our example, we have tags (discussed next), the ports (the port numbers it should activate on), the name and description -- which are all useful.

Tagging System

The tagging system is a revelation in choosing scripts to run.

Instead of creating pre-defined groups (such as "quiet" scripts), because we are letting users create and uploda their own scripts, we should use a tagging system.

Each module has "tags" as seen here tags = [core_approved, http, robotstxt, hyper, get_request].

These tags define what the module does, or other things about it. Think of it as tagging blogposts or categorising YouTube videos.

The creators of RustScan will have to decide on some pre-set tags, and should allow users to add their own tags if we see fit. As in, a user can submit a potential tag for use in the RustScan ecosystem.

A good starting tag is core_approved. This means the script is directly approved by the core team of RustScan for use. While we should vet every script we get, some scripts are easily more vetted than others.

Now, the question arises:

How do users enter tags?

And this is the mind-blowing part.

The tagging system is set theory.

Users can create intersections, unions and all sorts of set theory to personalise the category to exactly what they want. If you want everything in core_approved and quiet but is also in (intersecion) with http you can.

Set theory will allow you to construct any groups you want. Complete and utter freedom to the user.

But, the axiom of choice may come into play. There may be too many options for the user.

To counter this, we should create some set-categories which are just Set Theory but abstracted to an argument.

We can have an example for "http scripts approved by core_team", which in set theory is http and core_team, and then we can abstract it to a single argument such as -sHC for "H = HTTP" and "C = Core_team".

We let the users have the cake (complete and utter freedom) and eat it too (abstraction of arguments).

Example Tags

• Quiet
• Approved by core (that way we can vet scripts)
• HTTP
• Port number specific tags
• Service specific tags (this relies on another script to identify services running on ports)
• CTf (stuff like anonymous samba or ftp)

The RustScan Community Repo

These scripts should be stored in a repository. I propose we build a "RustScan Community Repo" for all of these scripts.

That way, we can easily control and handle the scripts themselves while also letting users have a centralised place for them if they wish to upload the scripts (not all scripts have to be uploaded)

Automated installation script in RustScan

When the user wants to use scripts, we can download the scripts from the GitHub repo (with an argument to allow downloading / updating) and store it somewhere like APPDIRS.

RustScan will then use this location of scripts to fuel its RSE (RustScan Scripting Engine).

If the scripts grow too large, we will eventually have to do something about that. But that's a problem for later us.

Applying this search to set theory.

Not only should users have the ability to use set theory, but we should build a tool which fuzzy-searches modules (so users can easily find them) and to run them if they wish.

An example of this is if the user was to search "http robots.txt". It would show all modules which matched this (via the description or name or tags), and the user can use these searches to execute scripts too.

If a user was to enter a specific name such as "http_robots_txt", the search will only return that so the user can have pin-point specific script usage.

"Fuzzy finding" scripts. Each script has a description, find scripts that match this description to run.

Heavy CI

For users to be able to submit scripts to RustScan, we should implement heavy continuous integration to make sure it doesn't break anything and to allow the core team to spend more time working on RustScan rather than the scripting engine.

[inainazumakun]

I think, we should also establish some ground rules for integrating the scripting engine into RustScan's core systems, and give a lot of freedom/flexibility to the scripter. Ideally, we should have three triggers for scripts set up, timed at:

1. before individual port scan
2. after individual port scan
3. post full scan

Ideally, in order to make sure RustScan stays lightweight and quick, prior to starting a scanning process, we should do all checks to cache a function call to a master controller class, so that way each individual port scan won't have to query through the entire scripting database. Additionally, we should allow parameter conditionals for scripts to optimize caching, i.e. scripts that trigger prior to port scan could state that they should only run on port 22 for SSH based scripts, or a post individual port-scan should have keywords for services found on a port.

example for clarification:
Instead of having each scanner process "check all scripts to see which scripts run" on the spot, we should just create a list that looks something like ["script1()", "script2()",] prior to scanning and pass this array onto the scanner processes, so that way each process is lightweight and doesn't contain any logic aside from the script call.
The list should also be different based on each script's requirement, i.e. scripts that only run between ports 100-500 should be put into this cache, and passed to the scanner processes working on ports 100-500.

For the master controller class- this class should be a multi-threadable worker that processes are assigned to, and is where the script call is actually executed from. This allows RustScan to execute multiple scripts at once, instead of trying to run scripts one-by-one on each port. The scanner process should ideally be looping through its script cache like this:

# prior to scan- script_cache should be generated by logic that checks if the scripts should be ran or not

script_cache = ["script1()", "script2()"]

script_master = script_master()

# script_master should be however many threads dedicated to running scripts on ports
# the master class should be the one actually performing the script calls, and will work on each assigned port one by one

# in the actual scanner process
if port == "open":
   for script in script_cache:

       # a function for caching a queue of scripts is required due to concurrency nature of the program, the master
       # would ideally work through this queue one by one until it's finished/idle
       script_master.queue(script)

[Nexus2k]

Your proposal sounds a lot like https://github.com/google/tsunami-security-scanner
How about integrating Rustscan into it?

[bergabman]

This is something new for me, and I'm not sure how deep we want to go into python, but I think the most maintainable would be a minimal connection with foreign languages.

I can really think of 2 things now.

1. build a complete scripting engine with that Pyo3 or similar, make the binding to all the stuff python scripts can use and let the users write specific scripts for RustScan. And make RustScan interpret the script and run it inside the binary.

2. make a thin wrapper on a python install available on the system of the user, (we can easily check for that) and the prerequisite of the python module/script would be to accept the output of RustScan which we have to decide on. Like this the users could make it sure that the script works just by running it manually then use it with RustScan. This would also help in the development of modules/scripts as it will not rely on RustScan.

I would rather have a RustScan which can just use python scripts, (or eventually any kind of scripts like bash or perl) than an interpreter in the binary that makes me rewrite my python script in a specific scripting way.
There are built in Command possibilities in Rust which is battle tested. You can rely on the execution and the return values. Basically you handle OS return values and actions. https://doc.rust-lang.org/std/process/struct.Command.html

There are also other libraris we can use for managing commands and subprocesses for execution.
https://crates.io/crates/subprocess

use subprocess::{Exec, ExitStatus};
fn execute_args(got_command: &str, args: &[&str]) -> Result<String, String> {
    let process = Exec::cmd(got_command).args(args);
    match process.capture() {
        Ok(c) => {
            let es = match c.exit_status {
                ExitStatus::Exited(c) => c as i32,
                ExitStatus::Signaled(c) => c as i32,
                ExitStatus::Other(c) => c,
                _ => -1,
            };
            if es != 0 {
                return Err(format!("exit code = {}", es));
            }
            Ok(c.stdout_str())
        }
        Err(error) => Err(error.to_string())
    }
}

This kind of thin wrapper can be easily maintained and improved in case we need improvements. The user will have to make it sure the python/perl/bash or any kind of interpreter is available on the system and that the script runs with it. From there on, we can be sure RustScan can use it as well.