Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

contains operator #99

Merged
merged 46 commits into from
Feb 27, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
46 commits
Select commit Hold shift + click to select a range
20fe19a
mark areas to support byte operator in filter string
diana-qing Jan 21, 2025
bef16a8
update byte rule
diana-qing Jan 21, 2025
8df748b
modify byte to just be one byte
diana-qing Jan 21, 2025
4b7e21f
treat Byte as a vector of bytes
diana-qing Jan 28, 2025
11781bd
use debug format for displaying Vec<u8>
diana-qing Jan 28, 2025
5389bfb
fix build issue
diana-qing Jan 28, 2025
173dee7
fix parsing of byte lit
diana-qing Feb 4, 2025
7c7890f
fix byte match: parse rhs.as_str() directly. convert proto.field to b…
diana-qing Feb 4, 2025
bb0050e
test multisubscription for byte matching
diana-qing Feb 4, 2025
a0fe1c4
log matching tls handshakes to file
diana-qing Feb 4, 2025
367b3af
update dependencies in Cargo.toml
diana-qing Feb 4, 2025
ba08336
run cargo expand for byte_match to view autogenerated code
diana-qing Feb 4, 2025
03748a1
delete commented out code
diana-qing Feb 4, 2025
9cca865
test multisub in same callback
diana-qing Feb 4, 2025
6618a6a
remove comments
diana-qing Feb 4, 2025
c827e69
Merge branch 'byte-op' of github.com:diana-qing/retina into byte-op
diana-qing Feb 4, 2025
4148289
use silent modifier
diana-qing Feb 4, 2025
c54628e
remove modifier
diana-qing Feb 4, 2025
4e14510
clean up code
diana-qing Feb 4, 2025
df83183
cargo fmt
diana-qing Feb 4, 2025
b521cbf
fix clippy warning
diana-qing Feb 4, 2025
85ba991
add contains op and implement for bytes
diana-qing Feb 11, 2025
1fc287d
fix build issues and test contains with bytes
diana-qing Feb 11, 2025
25ed1cb
implement contains for strings. test multisub for contains
diana-qing Feb 11, 2025
87616f5
Merge branch 'main' into contains-op
diana-qing Feb 18, 2025
903bb37
add logic for determining parent-child relationships when using contains
diana-qing Feb 18, 2025
5a60ea4
add unit tests to test is_excl and is_child when using contains
diana-qing Feb 18, 2025
88e3f5e
fix is_excl logic
diana-qing Feb 18, 2025
321925d
todo: fix logic for mc
diana-qing Feb 18, 2025
9261eca
Merge branch 'main' into contains-op
diana-qing Feb 18, 2025
94d4255
move contains op optimization logic to a different branch
diana-qing Feb 18, 2025
94c05c2
add sub to log_ssh to test contains
diana-qing Feb 18, 2025
318f257
add contains as a comparison op in lib
diana-qing Feb 18, 2025
d02ef53
run cargo fmt
diana-qing Feb 18, 2025
8a2e9eb
add example for using contains with string
diana-qing Feb 18, 2025
f5ad411
fix ex's
diana-qing Feb 18, 2025
694485d
remove .DS_Store files
diana-qing Feb 18, 2025
82cf7c7
remove commented out example
diana-qing Feb 18, 2025
ad463fa
remove contains_test
diana-qing Feb 18, 2025
0a8b1a3
remove print statements from example
diana-qing Feb 18, 2025
043df3d
fix formatting in docs
diana-qing Feb 19, 2025
54fc635
undo spacing change in docs
diana-qing Feb 25, 2025
2fd2a17
user Finder to enable faster searching in bytes
diana-qing Feb 25, 2025
3b7021e
use Finder for substring search since contains can be slow for large …
diana-qing Feb 25, 2025
b0ddc49
use static to avoid recreating Finder every time
diana-qing Feb 26, 2025
af22b63
run fmt
diana-qing Feb 26, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion core/src/filter/ast.rs
Original file line number Diff line number Diff line change
Expand Up @@ -667,7 +667,7 @@ pub(super) fn is_excl_int(
BinOp::In => return peer_to < from || peer_from > to,
_ => {}
},
BinOp::Re | BinOp::En => {}
BinOp::Re | BinOp::En | BinOp::Contains => {}
}
false
}
Expand Down Expand Up @@ -802,6 +802,7 @@ pub enum BinOp {
In,
Re,
En,
Contains,
}

impl fmt::Display for BinOp {
Expand All @@ -816,6 +817,7 @@ impl fmt::Display for BinOp {
BinOp::In => write!(f, "in"),
BinOp::Re => write!(f, "matches"),
BinOp::En => write!(f, "eq"),
BinOp::Contains => write!(f, "contains"),
}
}
}
Expand Down
3 changes: 2 additions & 1 deletion core/src/filter/grammar.pest
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,7 @@ not_op = { "!" | "not" | "NOT" } // not yet supported, temp placeholder
// Binary operators
// ----------------------------------------------------------------------
bin_op = {
eq_op | ne_op | ge_op | le_op | gt_op | lt_op | in_op | re_op | en_op
eq_op | ne_op | ge_op | le_op | gt_op | lt_op | in_op | re_op | en_op | contains_op
}

eq_op = { "=" }
Expand All @@ -65,6 +65,7 @@ lt_op = { "<" | "lt" }
in_op = { "in" }
re_op = { "~" | "matches" }
en_op = { "eq" }
contains_op = { "contains" }

// Miscellaneous
// ----------------------------------------------------------------------
Expand Down
1 change: 1 addition & 0 deletions core/src/filter/parser.rs
Original file line number Diff line number Diff line change
Expand Up @@ -183,6 +183,7 @@ impl FilterParser {
Rule::in_op => Ok(BinOp::In),
Rule::re_op => Ok(BinOp::Re),
Rule::en_op => Ok(BinOp::En),
Rule::contains_op => Ok(BinOp::Contains),
_ => bail!(FilterError::InvalidBinOp(op_str)),
}
}
Expand Down
3 changes: 2 additions & 1 deletion examples/log_ssh/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -11,4 +11,5 @@ retina-datatypes = { path = "../../datatypes" }
serde = { version = "1.0", features = ["derive"] }
serde_json = "1.0.96"
lazy_static = "1.4.0"
regex = "1.7.3"
regex = "1.7.3"
memchr = "2.7.4"
19 changes: 14 additions & 5 deletions examples/log_ssh/src/main.rs
Original file line number Diff line number Diff line change
Expand Up @@ -28,25 +28,34 @@ struct Args {
outfile: PathBuf,
}

#[filter("ssh.software_version_ctos ~ '^OpenSSH_[0-9]+\\.[0-9].*$'")]
fn ssh_cb(ssh: &SshHandshake) {
#[filter("ssh.protocol_version_ctos = |32 2E 30|")]
fn ssh_byte_match_cb(ssh: &SshHandshake) {
if let Ok(serialized) = serde_json::to_string(&ssh) {
let mut wtr = file.lock().unwrap();
wtr.write_all(serialized.as_bytes()).unwrap();
wtr.write_all(b"\n").unwrap();
}
}

#[filter("ssh.protocol_version_ctos = |32 2E 30|")]
fn ssh_byte_match_cb(ssh: &SshHandshake) {
#[filter("ssh.key_exchange_cookie_stoc contains |15 1A|")]
fn ssh_contains_bytes_cb(ssh: &SshHandshake) {
if let Ok(serialized) = serde_json::to_string(&ssh) {
let mut wtr = file.lock().unwrap();
wtr.write_all(serialized.as_bytes()).unwrap();
wtr.write_all(b"\n").unwrap();
}
}

#[filter("ssh.software_version_ctos contains 'OpenSSH'")]
fn ssh_contains_str_cb(ssh: &SshHandshake) {
if let Ok(serialized) = serde_json::to_string(&ssh) {
let mut wtr = file.lock().unwrap();
wtr.write_all(serialized.as_bytes()).unwrap();
wtr.write_all(b"\n").unwrap();
}
}

#[retina_main(2)]
#[retina_main(3)]
fn main() {
let args = Args::parse();
let config = load_config(&args.config);
Expand Down
3 changes: 2 additions & 1 deletion filtergen/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -19,4 +19,5 @@ serde_with = "1.6.1"
serde = { version = "1.0", features = ["derive"] }
toml = "0.5.11"
retina-datatypes = { path = "../datatypes" }
lazy_static = "1.4.0"
lazy_static = "1.4.0"
memchr = "2.7.4"
1 change: 1 addition & 0 deletions filtergen/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -151,6 +151,7 @@
//! | `<` | `lt` | Less than | `ipv6.payload_length < 500` |
//! | `in` | | In a range, or in a subnet | `ipv4.src_addr in 1.2.3.4/16` |
//! | `~` | `matches` | Regular expression match | `tls.sni ~ 'netflix\\.com$'` |
//! | `contains` | | Check if right appears in left | `ssh.key_exchange_cookie_stoc contains \|15 A1\|` |
//!
//! **Possible pitfalls involving `!=`**
//!
Expand Down
26 changes: 26 additions & 0 deletions filtergen/src/utils.rs
Original file line number Diff line number Diff line change
Expand Up @@ -152,6 +152,19 @@ pub(crate) fn binary_to_tokens(
// Regex::new(#val_lit).unwrap().is_match(#proto.#field())
// }
}
BinOp::Contains => {
let val_lit = syn::LitStr::new(text, Span::call_site());

let finder_name = format!("FINDER{}", statics.len());
let finder_ident = Ident::new(&finder_name, Span::call_site());
let lazy_finder = quote! {
static ref #finder_ident: memchr::memmem::Finder<'static> = memchr::memmem::Finder::new(#val_lit.as_bytes());
};
statics.push(lazy_finder);
quote! {
#finder_ident.find(#proto.#field().as_bytes()).is_some()
}
}
_ => panic!("Invalid binary operation `{}` for value: `{}`.", op, value),
}
}
Expand All @@ -162,6 +175,19 @@ pub(crate) fn binary_to_tokens(
#proto.#field().as_bytes() == #bytes_lit
}
}
BinOp::Contains => {
let bytes_lit = syn::LitByteStr::new(b, Span::call_site());

let finder_name = format!("FINDER{}", statics.len());
let finder_ident = Ident::new(&finder_name, Span::call_site());
let lazy_finder = quote! {
static ref #finder_ident: memchr::memmem::Finder<'static> = memchr::memmem::Finder::new(#bytes_lit);
};
statics.push(lazy_finder);
quote! {
#finder_ident.find(#proto.#field().as_ref()).is_some()
}
}
_ => panic!("Invalid binary operation `{}` for value: `{}`.", op, value),
},
}
Expand Down
Loading