From ea3524c074f0d7153f03f648cfee717b8690f50a Mon Sep 17 00:00:00 2001 From: Grzegorz Nosek Date: Wed, 18 Aug 2021 15:21:35 +0200 Subject: [PATCH 001/148] new(proposals): API versioning for user/kernel boundary Signed-off-by: Grzegorz Nosek Add a link to https://semver.org/ Co-authored-by: Angelo Puglisi --- proposals/20210818-driver-semver.md | 95 +++++++++++++++++++++++++++++ 1 file changed, 95 insertions(+) create mode 100644 proposals/20210818-driver-semver.md diff --git a/proposals/20210818-driver-semver.md b/proposals/20210818-driver-semver.md new file mode 100644 index 000000000..0039323d1 --- /dev/null +++ b/proposals/20210818-driver-semver.md @@ -0,0 +1,95 @@ +# API versioning for user/kernel boundary + +## Summary + +This proposal introduces [semver](https://semver.org/)-compatible version checks for the user/kernel boundary, +i.e. between the kernel driver/eBPF probe and the userspace components. + +Currently, to ensure compatibility, the kernel module/eBPF probe must be built together +with libscap. Even though actual incompatibilities are few and far between, there's no +mechanism to tell whether a particular kernel module/eBPF probe is new enough to work +with a particular libscap build. + +The version checks at present are: + +1. For the eBPF probe, an exact match of the version of the probe/userspace components +is required. The probe version is effectively the version number of the libscap *consumer*, +not directly related to libscap itself. E.g. two different consumer releases can use +the same libscap commit and still be unable to share the probes. + +2. For the kernel module, there is no version check at all. The driver exposes an ioctl +to get the probe version (`PPM_IOCTL_GET_PROBE_VERSION`) but it is not used anywhere. +Again, what is versioned is the libscap consumer, not actual libscap. + +## Motivation + +Introducing a machine-usable API versioning scheme will let us: + +1. Cut down the number of probes needed to be built. Instead of a probe for each +(kernel version, consumer name, consumer version) tuple, we would only need one +for each (kernel version, driver API version). Given the relatively slow development +on the kernel side, a single driver API version might live for a long time + +2. Make upgrades easier. Currently, the probe has to be unloaded and a new one loaded +in its place. With the API versioning scheme in place, usually there won't even +be a different probe. If there is one, it will usually be forwards-compatible, +even though it might miss some bug fixes. Only if the versions are truly incompatible, +an unload/reload will be required. Note that the API version can live in the module +*name* itself, which would let us have multiple versions loaded at the same time, +if we decide to go that way. + +3. Ship the probes in Linux distributions. With a single consumer-agnostic probe +per kernel, it becomes realistic for Linux distributions to ship a prebuilt probe +for their kernels. That would make all libscap consumers work out of the box, +without worrying about shipping the probe. + +4. Support older consumer versions with new kernels. Whenever a new kernel comes out, +probes need to be built to support it. These probes currently cannot be used +for older consumer releases, even though there are no technical issues that would +prevent it. + +## Goals + +* Make the probes reusable across libscap consumers and their versions + +## Non-goals + +* Make the probes reusable across kernel versions (this is probably impossible + in a general way for the kernel module, but BTF/CO-RE may help for the eBPF + probe). + +## The plan + +1. Introduce an API version embedded in the userspace and kernel code + * The version number will be a single 64-bit number that can be decomposed + to the three semantic versioning components (major, minor, patch) + +2. Extend the review process to ensure the API version is incremented when needed. + Note that e.g. adding support for a new kernel should not result in an API + version increase (if the probe failed to build for that particular kernel + before). + +3. Verify the API version of the kernel module/eBPF probe when starting + the libscap consumer: + * different major versions cause a hard error + * kernel minor < userspace minor causes a hard error + * kernel minor == userspace minor and kernel patch < userspace patch causes + a warning (the probe is compatible but has known bugs, fixed in later + versions) + +4. Remove consumer name and version from the libscap build process + * The probe should always be named `scap_probe` and use the API version to identify + itself + +## The non-plan + +This proposal does not address changes to the infrastructure that consumer +projects may have to build and distribute probes. To fully realize the benefits +of this proposal, such infrastructure would have to be adapted to use e.g. +`scap_probe--` as the identifier for a particular +probe, instead of `_probe--`. + +Since the last point of the plan changes the module name, consumers having +said infrastructure will have to make these changes before upgrading +to a libscap version implementing that point. We might devise a plan to smooth +the transition (e.g. allow building the probe under both names for a while). From afb13cf3be64ae7af92b97e0e39e691c1cfaf67e Mon Sep 17 00:00:00 2001 From: Grzegorz Nosek Date: Mon, 23 Aug 2021 17:21:32 +0200 Subject: [PATCH 002/148] Apply suggestions from code review Signed-off-by: Grzegorz Nosek Co-authored-by: Leonardo Grasso --- proposals/20210818-driver-semver.md | 26 +++++++++++++------------- 1 file changed, 13 insertions(+), 13 deletions(-) diff --git a/proposals/20210818-driver-semver.md b/proposals/20210818-driver-semver.md index 0039323d1..08dd3692f 100644 --- a/proposals/20210818-driver-semver.md +++ b/proposals/20210818-driver-semver.md @@ -25,36 +25,36 @@ Again, what is versioned is the libscap consumer, not actual libscap. Introducing a machine-usable API versioning scheme will let us: -1. Cut down the number of probes needed to be built. Instead of a probe for each +1. Cut down the number of drivers needed to be built. Instead of a driver for each (kernel version, consumer name, consumer version) tuple, we would only need one for each (kernel version, driver API version). Given the relatively slow development on the kernel side, a single driver API version might live for a long time -2. Make upgrades easier. Currently, the probe has to be unloaded and a new one loaded +2. Make upgrades easier. Currently, the driver has to be unloaded and a new one loaded in its place. With the API versioning scheme in place, usually there won't even -be a different probe. If there is one, it will usually be forwards-compatible, +be a different driver. If there is one, it will usually be forwards-compatible, even though it might miss some bug fixes. Only if the versions are truly incompatible, an unload/reload will be required. Note that the API version can live in the module *name* itself, which would let us have multiple versions loaded at the same time, if we decide to go that way. -3. Ship the probes in Linux distributions. With a single consumer-agnostic probe -per kernel, it becomes realistic for Linux distributions to ship a prebuilt probe +3. Ship the drivers in Linux distributions. With a single consumer-agnostic driver +per kernel, it becomes realistic for Linux distributions to ship a prebuilt driver for their kernels. That would make all libscap consumers work out of the box, -without worrying about shipping the probe. +without worrying about shipping the driver. 4. Support older consumer versions with new kernels. Whenever a new kernel comes out, -probes need to be built to support it. These probes currently cannot be used +drivers need to be built to support it. These drivers currently cannot be used for older consumer releases, even though there are no technical issues that would prevent it. ## Goals -* Make the probes reusable across libscap consumers and their versions +* Make the drivers reusable across libscap consumers and their versions ## Non-goals -* Make the probes reusable across kernel versions (this is probably impossible +* Make the drivers reusable across kernel versions (this is probably impossible in a general way for the kernel module, but BTF/CO-RE may help for the eBPF probe). @@ -66,7 +66,7 @@ prevent it. 2. Extend the review process to ensure the API version is incremented when needed. Note that e.g. adding support for a new kernel should not result in an API - version increase (if the probe failed to build for that particular kernel + version increase (if the driver failed to build for that particular kernel before). 3. Verify the API version of the kernel module/eBPF probe when starting @@ -74,7 +74,7 @@ prevent it. * different major versions cause a hard error * kernel minor < userspace minor causes a hard error * kernel minor == userspace minor and kernel patch < userspace patch causes - a warning (the probe is compatible but has known bugs, fixed in later + a warning (the driver is compatible but has known bugs, fixed in later versions) 4. Remove consumer name and version from the libscap build process @@ -84,7 +84,7 @@ prevent it. ## The non-plan This proposal does not address changes to the infrastructure that consumer -projects may have to build and distribute probes. To fully realize the benefits +projects may have to build and distribute drivers. To fully realize the benefits of this proposal, such infrastructure would have to be adapted to use e.g. `scap_probe--` as the identifier for a particular probe, instead of `_probe--`. @@ -92,4 +92,4 @@ probe, instead of `_probe--`. Since the last point of the plan changes the module name, consumers having said infrastructure will have to make these changes before upgrading to a libscap version implementing that point. We might devise a plan to smooth -the transition (e.g. allow building the probe under both names for a while). +the transition (e.g. allow building the driver under both names for a while). From 92d3cd41e094b0b133c63593492635bf19dcb60e Mon Sep 17 00:00:00 2001 From: Grzegorz Nosek Date: Mon, 23 Aug 2021 17:49:03 +0200 Subject: [PATCH 003/148] Review comments Signed-off-by: Grzegorz Nosek --- proposals/20210818-driver-semver.md | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/proposals/20210818-driver-semver.md b/proposals/20210818-driver-semver.md index 08dd3692f..52f97bffd 100644 --- a/proposals/20210818-driver-semver.md +++ b/proposals/20210818-driver-semver.md @@ -63,6 +63,10 @@ prevent it. 1. Introduce an API version embedded in the userspace and kernel code * The version number will be a single 64-bit number that can be decomposed to the three semantic versioning components (major, minor, patch) + * As long as the API version is kept separate from any preexisting + consumer version numbers, the API version can start at 1.0.0. + The easiest way to accomplish this would be to rename the driver + (step 4). 2. Extend the review process to ensure the API version is incremented when needed. Note that e.g. adding support for a new kernel should not result in an API @@ -77,9 +81,11 @@ prevent it. a warning (the driver is compatible but has known bugs, fixed in later versions) -4. Remove consumer name and version from the libscap build process - * The probe should always be named `scap_probe` and use the API version to identify +4. Deemphasize consumer name and version from the libscap build process + * The driver should be named `scap` by default and use the API version to identify itself + * An option may remain to override the driver name (and supply a version) + but it should not be used without good reason ## The non-plan From 89e1263908a36abb55a113813076d9983c9c1cfa Mon Sep 17 00:00:00 2001 From: "Shang-Wen Wang (Sam Wang)" Date: Thu, 2 Sep 2021 00:21:04 -0400 Subject: [PATCH 004/148] fix: container.privileged in containerd and crio Retrieve privileged from the correct locations of ContainerStatusResponse. - containerd: info/config/linux/security_context/privileged - crio: info/privileged Signed-off-by: Shang-Wen Wang (Sam Wang) --- userspace/libsinsp/container_engine/cri.cpp | 2 +- userspace/libsinsp/cri.cpp | 17 ++++++++++++++++- userspace/libsinsp/cri.h | 4 +--- 3 files changed, 18 insertions(+), 5 deletions(-) diff --git a/userspace/libsinsp/container_engine/cri.cpp b/userspace/libsinsp/container_engine/cri.cpp index 59ccb2a9c..22fcb5f3d 100644 --- a/userspace/libsinsp/container_engine/cri.cpp +++ b/userspace/libsinsp/container_engine/cri.cpp @@ -71,7 +71,7 @@ bool cri_async_source::parse_containerd(const runtime::v1alpha2::ContainerStatus m_cri->parse_cri_env(root, container); m_cri->parse_cri_json_image(root, container); - bool ret = m_cri->parse_cri_runtime_spec(root, container); + bool ret = m_cri->parse_cri_ext_container_info(root, container); if(root.isMember("sandboxID") && root["sandboxID"].isString()) { diff --git a/userspace/libsinsp/cri.cpp b/userspace/libsinsp/cri.cpp index 12b3ae6ca..dab68f903 100644 --- a/userspace/libsinsp/cri.cpp +++ b/userspace/libsinsp/cri.cpp @@ -277,7 +277,7 @@ bool cri_interface::parse_cri_json_image(const Json::Value &info, sinsp_containe return true; } -bool cri_interface::parse_cri_runtime_spec(const Json::Value &info, sinsp_container_info &container) +bool cri_interface::parse_cri_ext_container_info(const Json::Value &info, sinsp_container_info &container) { const Json::Value *linux = nullptr; if(!walk_down_json(info, &linux, "runtimeSpec", "linux") || !linux->isObject()) @@ -301,10 +301,25 @@ bool cri_interface::parse_cri_runtime_spec(const Json::Value &info, sinsp_contai set_numeric_32(*cpu, "cpuset_cpu_count", container.m_cpuset_cpu_count); } + bool priv_found = false; const Json::Value *privileged; + // old containerd? if(walk_down_json(*linux, &privileged, "security_context", "privileged") && privileged->isBool()) { container.m_privileged = privileged->asBool(); + priv_found = true; + } + + // containerd + if(!priv_found && walk_down_json(info, &privileged, "config", "linux", "security_context", "privileged") && privileged->isBool()) { + container.m_privileged = privileged->asBool(); + priv_found = true; + } + + // cri-o + if(!priv_found && walk_down_json(info, &privileged, "privileged") && privileged->isBool()) { + container.m_privileged = privileged->asBool(); + priv_found = true; } return true; diff --git a/userspace/libsinsp/cri.h b/userspace/libsinsp/cri.h index 49cd5c6fd..d5e278651 100644 --- a/userspace/libsinsp/cri.h +++ b/userspace/libsinsp/cri.h @@ -120,10 +120,8 @@ class cri_interface * @param info the `info` key of the `info` field of the ContainerStatusResponse * @param container the container info to fill out * @return true if successful - * - * Note: only containerd exposes this data */ - bool parse_cri_runtime_spec(const Json::Value &info, sinsp_container_info &container); + bool parse_cri_ext_container_info(const Json::Value &info, sinsp_container_info &container); /** * @brief check if the passed container ID is a pod sandbox (pause container) From 1e9c3c73d373a7f9cf414b72b792c5d13663e6e4 Mon Sep 17 00:00:00 2001 From: lucklypse Date: Fri, 17 Sep 2021 15:34:09 +0000 Subject: [PATCH 005/148] cmake: add curl as a sinsp dependency Signed-off-by: lucklypse --- cmake/modules/libsinsp.cmake | 3 +++ 1 file changed, 3 insertions(+) diff --git a/cmake/modules/libsinsp.cmake b/cmake/modules/libsinsp.cmake index 578fec479..9b184c5ce 100644 --- a/cmake/modules/libsinsp.cmake +++ b/cmake/modules/libsinsp.cmake @@ -19,6 +19,7 @@ endif() include(jsoncpp) if(NOT MINIMAL_BUILD) include(cares) + include(curl) endif() set(LIBSINSP_INCLUDE_DIRS ${LIBSINSP_DIR}/userspace/libsinsp ${LIBSINSP_DIR}/common ${LIBSCAP_INCLUDE_DIRS} ${DRIVER_CONFIG_DIR}) @@ -40,6 +41,8 @@ list(APPEND LIBSINSP_INCLUDE_DIRS ${JSONCPP_ABSOLUTE_INCLUDE_DIR}) if(NOT MINIMAL_BUILD) get_filename_component(CARES_ABSOLUTE_INCLUDE_DIR ${CARES_INCLUDE} ABSOLUTE) list(APPEND LIBSINSP_INCLUDE_DIRS ${CARES_ABSOLUTE_INCLUDE_DIR}) + get_filename_component(CURL_ABSOLUTE_INCLUDE_DIR ${CURL_INCLUDE_DIR} ABSOLUTE) + list(APPEND LIBSINSP_INCLUDE_DIRS ${CURL_ABSOLUTE_INCLUDE_DIR}) endif() add_subdirectory(${LIBSINSP_DIR}/userspace/libsinsp ${CMAKE_BINARY_DIR}/libsinsp) From a05c330db11e0e80b239d09d6b83a8beae760a61 Mon Sep 17 00:00:00 2001 From: Federico Di Pierro Date: Mon, 13 Sep 2021 17:02:09 +0200 Subject: [PATCH 006/148] fix(driver): fix bpf probe verifier for bpf_poll_parse_fds() and bpf_parse_readv_writev_bufs() on recent kernel and llvm (linux 5.14.2 and llvm 12.0.1). Signed-off-by: Federico Di Pierro Co-authored-by: Jason Dellaluce --- driver/bpf/fillers.h | 15 +++++++-------- 1 file changed, 7 insertions(+), 8 deletions(-) diff --git a/driver/bpf/fillers.h b/driver/bpf/fillers.h index 791d0fb29..7a5cd07b5 100644 --- a/driver/bpf/fillers.h +++ b/driver/bpf/fillers.h @@ -308,26 +308,21 @@ static __always_inline int bpf_poll_parse_fds(struct filler_data *data, #pragma unroll for (j = 0; j < POLL_MAXFDS; ++j) { - u16 flags; + if (off > SCRATCH_SIZE_HALF) + return PPM_FAILURE_BUFFER_FULL; if (j == nfds) break; + u16 flags; if (enter_event) { flags = poll_events_to_scap(fds[j].events); } else { - if (!fds[j].revents) - continue; - flags = poll_events_to_scap(fds[j].revents); } - if (off > SCRATCH_SIZE_HALF) - return PPM_FAILURE_BUFFER_FULL; - *(s64 *)&data->buf[off & SCRATCH_SIZE_HALF] = fds[j].fd; off += sizeof(s64); - if (off > SCRATCH_SIZE_HALF) return PPM_FAILURE_BUFFER_FULL; @@ -415,10 +410,14 @@ static __always_inline int bpf_parse_readv_writev_bufs(struct filler_data *data, #endif return PPM_FAILURE_INVALID_USER_MEMORY; + #pragma unroll for (j = 0; j < MAX_IOVCNT; ++j) { if (j == iovcnt) break; + // BPF seems to require an hard limit to avoid overflows + if (size == LONG_MAX) + break; size += iov[j].iov_len; } From 8cde1840166814438276b08a6be48b45cb63e37d Mon Sep 17 00:00:00 2001 From: Federico Di Pierro Date: Tue, 14 Sep 2021 14:54:54 +0200 Subject: [PATCH 007/148] Update driver/bpf/fillers.h Signed-off-by: Federico Di Pierro Co-authored-by: Michele Zuccala --- driver/bpf/fillers.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/driver/bpf/fillers.h b/driver/bpf/fillers.h index 7a5cd07b5..28541b332 100644 --- a/driver/bpf/fillers.h +++ b/driver/bpf/fillers.h @@ -415,7 +415,7 @@ static __always_inline int bpf_parse_readv_writev_bufs(struct filler_data *data, for (j = 0; j < MAX_IOVCNT; ++j) { if (j == iovcnt) break; - // BPF seems to require an hard limit to avoid overflows + // BPF seems to require a hard limit to avoid overflows if (size == LONG_MAX) break; From 747934f75e3c497403c4d5ee51d3fbb373076575 Mon Sep 17 00:00:00 2001 From: Jason Dellaluce Date: Thu, 16 Sep 2021 08:44:34 +0000 Subject: [PATCH 008/148] fix(driver/bpf): fix verifier issues in latest clang versions Signed-off-by: Jason Dellaluce --- driver/bpf/filler_helpers.h | 39 ++++++++++++++++++++----------------- driver/bpf/fillers.h | 18 ++++++++++++----- 2 files changed, 34 insertions(+), 23 deletions(-) diff --git a/driver/bpf/filler_helpers.h b/driver/bpf/filler_helpers.h index 147b4a059..b4f9456f9 100644 --- a/driver/bpf/filler_helpers.h +++ b/driver/bpf/filler_helpers.h @@ -719,17 +719,20 @@ static __always_inline int __bpf_val_to_ring(struct filler_data *data, { unsigned int len_dyn = 0; unsigned int len; + unsigned long curoff_bounded; + curoff_bounded = data->state->tail_ctx.curoff & SCRATCH_SIZE_HALF; if (data->state->tail_ctx.curoff > SCRATCH_SIZE_HALF) return PPM_FAILURE_BUFFER_FULL; - + if (dyn_idx != (u8)-1) { - *((u8 *)&data->buf[data->state->tail_ctx.curoff & SCRATCH_SIZE_HALF]) = dyn_idx; + *((u8 *)&data->buf[curoff_bounded]) = dyn_idx; len_dyn = sizeof(u8); data->state->tail_ctx.curoff += len_dyn; data->state->tail_ctx.len += len_dyn; } + curoff_bounded = data->state->tail_ctx.curoff & SCRATCH_SIZE_HALF; if (data->state->tail_ctx.curoff > SCRATCH_SIZE_HALF) return PPM_FAILURE_BUFFER_FULL; @@ -740,7 +743,7 @@ static __always_inline int __bpf_val_to_ring(struct filler_data *data, if (!data->curarg_already_on_frame) { int res; - res = bpf_probe_read_str(&data->buf[data->state->tail_ctx.curoff & SCRATCH_SIZE_HALF], + res = bpf_probe_read_str(&data->buf[curoff_bounded], PPM_MAX_ARG_SIZE, (const void *)val); if (res == -EFAULT) @@ -763,15 +766,15 @@ static __always_inline int __bpf_val_to_ring(struct filler_data *data, dpi_lookahead_size = len; if (!data->curarg_already_on_frame) { - volatile unsigned long read_size = dpi_lookahead_size; + volatile u16 read_size = dpi_lookahead_size; #ifdef BPF_FORBIDS_ZERO_ACCESS if (read_size) - if (bpf_probe_read(&data->buf[data->state->tail_ctx.curoff & SCRATCH_SIZE_HALF], + if (bpf_probe_read(&data->buf[curoff], ((read_size - 1) & SCRATCH_SIZE_HALF) + 1, (void *)val)) #else - if (bpf_probe_read(&data->buf[data->state->tail_ctx.curoff & SCRATCH_SIZE_HALF], + if (bpf_probe_read(&data->buf[curoff_bounded], read_size & SCRATCH_SIZE_HALF, (void *)val)) #endif @@ -787,15 +790,15 @@ static __always_inline int __bpf_val_to_ring(struct filler_data *data, len = PPM_MAX_ARG_SIZE; if (!data->curarg_already_on_frame) { - volatile unsigned long read_size = len; + volatile u16 read_size = len; #ifdef BPF_FORBIDS_ZERO_ACCESS if (read_size) - if (bpf_probe_read(&data->buf[data->state->tail_ctx.curoff & SCRATCH_SIZE_HALF], + if (bpf_probe_read(&data->buf[curoff], ((read_size - 1) & SCRATCH_SIZE_HALF) + 1, (void *)val)) #else - if (bpf_probe_read(&data->buf[data->state->tail_ctx.curoff & SCRATCH_SIZE_HALF], + if (bpf_probe_read(&data->buf[curoff_bounded], read_size & SCRATCH_SIZE_HALF, (void *)val)) #endif @@ -821,13 +824,13 @@ static __always_inline int __bpf_val_to_ring(struct filler_data *data, case PT_FLAGS8: case PT_UINT8: case PT_SIGTYPE: - *((u8 *)&data->buf[data->state->tail_ctx.curoff & SCRATCH_SIZE_HALF]) = val; + *((u8 *)&data->buf[curoff_bounded]) = val; len = sizeof(u8); break; case PT_FLAGS16: case PT_UINT16: case PT_SYSCALLID: - *((u16 *)&data->buf[data->state->tail_ctx.curoff & SCRATCH_SIZE_HALF]) = val; + *((u16 *)&data->buf[curoff_bounded]) = val; len = sizeof(u16); break; case PT_FLAGS32: @@ -836,32 +839,32 @@ static __always_inline int __bpf_val_to_ring(struct filler_data *data, case PT_UID: case PT_GID: case PT_SIGSET: - *((u32 *)&data->buf[data->state->tail_ctx.curoff & SCRATCH_SIZE_HALF]) = val; + *((u32 *)&data->buf[curoff_bounded]) = val; len = sizeof(u32); break; case PT_RELTIME: case PT_ABSTIME: case PT_UINT64: - *((u64 *)&data->buf[data->state->tail_ctx.curoff & SCRATCH_SIZE_HALF]) = val; + *((u64 *)&data->buf[curoff_bounded]) = val; len = sizeof(u64); break; case PT_INT8: - *((s8 *)&data->buf[data->state->tail_ctx.curoff & SCRATCH_SIZE_HALF]) = val; + *((s8 *)&data->buf[curoff_bounded]) = val; len = sizeof(s8); break; case PT_INT16: - *((s16 *)&data->buf[data->state->tail_ctx.curoff & SCRATCH_SIZE_HALF]) = val; + *((s16 *)&data->buf[curoff_bounded]) = val; len = sizeof(s16); break; case PT_INT32: - *((s32 *)&data->buf[data->state->tail_ctx.curoff & SCRATCH_SIZE_HALF]) = val; + *((s32 *)&data->buf[curoff_bounded]) = val; len = sizeof(s32); break; case PT_INT64: case PT_ERRNO: case PT_FD: case PT_PID: - *((s64 *)&data->buf[data->state->tail_ctx.curoff & SCRATCH_SIZE_HALF]) = val; + *((s64 *)&data->buf[curoff_bounded]) = val; len = sizeof(s64); break; default: { @@ -871,7 +874,7 @@ static __always_inline int __bpf_val_to_ring(struct filler_data *data, return PPM_FAILURE_BUG; } } - + if (len_dyn + len > PPM_MAX_ARG_SIZE) return PPM_FAILURE_BUFFER_FULL; diff --git a/driver/bpf/fillers.h b/driver/bpf/fillers.h index 28541b332..4fe0665c9 100644 --- a/driver/bpf/fillers.h +++ b/driver/bpf/fillers.h @@ -1505,11 +1505,13 @@ static __always_inline int __bpf_append_cgroup(struct css_set *cgroups, char *cgroup_path[MAX_CGROUP_PATHS]; bool prev_empty = false; int off = *len; + unsigned int off_bounded; + off_bounded = off & SCRATCH_SIZE_HALF; if (off > SCRATCH_SIZE_HALF) return PPM_FAILURE_BUFFER_FULL; - int res = bpf_probe_read_str(&buf[off & SCRATCH_SIZE_HALF], + int res = bpf_probe_read_str(&buf[off_bounded], SCRATCH_SIZE_HALF, subsys_name); if (res == -EFAULT) @@ -1517,11 +1519,13 @@ static __always_inline int __bpf_append_cgroup(struct css_set *cgroups, off += res - 1; + off_bounded = off & SCRATCH_SIZE_HALF; if (off > SCRATCH_SIZE_HALF) return PPM_FAILURE_BUFFER_FULL; - buf[off & SCRATCH_SIZE_HALF] = '='; + buf[off_bounded] = '='; ++off; + off_bounded = off & SCRATCH_SIZE_HALF; #pragma unroll MAX_CGROUP_PATHS for (int k = 0; k < MAX_CGROUP_PATHS; ++k) { @@ -1540,8 +1544,9 @@ static __always_inline int __bpf_append_cgroup(struct css_set *cgroups, if (off > SCRATCH_SIZE_HALF) return PPM_FAILURE_BUFFER_FULL; - buf[off & SCRATCH_SIZE_HALF] = '/'; + buf[off_bounded] = '/'; ++off; + off_bounded = off & SCRATCH_SIZE_HALF; } prev_empty = false; @@ -1549,11 +1554,14 @@ static __always_inline int __bpf_append_cgroup(struct css_set *cgroups, if (off > SCRATCH_SIZE_HALF) return PPM_FAILURE_BUFFER_FULL; - res = bpf_probe_read_str(&buf[off & SCRATCH_SIZE_HALF], + res = bpf_probe_read_str(&buf[off_bounded], SCRATCH_SIZE_HALF, cgroup_path[k]); if (res > 1) + { off += res - 1; + off_bounded = off & SCRATCH_SIZE_HALF; + } else if (res == 1) prev_empty = true; else @@ -1564,7 +1572,7 @@ static __always_inline int __bpf_append_cgroup(struct css_set *cgroups, if (off > SCRATCH_SIZE_HALF) return PPM_FAILURE_BUFFER_FULL; - buf[off & SCRATCH_SIZE_HALF] = 0; + buf[off_bounded] = 0; ++off; *len = off; From 8f5b5e8c61ceb06ec40207e4c4e0f98b4a8d9201 Mon Sep 17 00:00:00 2001 From: Jason Dellaluce Date: Thu, 16 Sep 2021 12:20:23 +0000 Subject: [PATCH 009/148] fix(driver/bpf): fixing typos Signed-off-by: Jason Dellaluce --- driver/bpf/filler_helpers.h | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/driver/bpf/filler_helpers.h b/driver/bpf/filler_helpers.h index b4f9456f9..8f440ac3e 100644 --- a/driver/bpf/filler_helpers.h +++ b/driver/bpf/filler_helpers.h @@ -770,7 +770,7 @@ static __always_inline int __bpf_val_to_ring(struct filler_data *data, #ifdef BPF_FORBIDS_ZERO_ACCESS if (read_size) - if (bpf_probe_read(&data->buf[curoff], + if (bpf_probe_read(&data->buf[curoff_bounded], ((read_size - 1) & SCRATCH_SIZE_HALF) + 1, (void *)val)) #else @@ -794,7 +794,7 @@ static __always_inline int __bpf_val_to_ring(struct filler_data *data, #ifdef BPF_FORBIDS_ZERO_ACCESS if (read_size) - if (bpf_probe_read(&data->buf[curoff], + if (bpf_probe_read(&data->buf[curoff_bounded], ((read_size - 1) & SCRATCH_SIZE_HALF) + 1, (void *)val)) #else From 515ea985ef850cdafcc8329fa68465b2a8c229ed Mon Sep 17 00:00:00 2001 From: Federico Di Pierro Date: Thu, 16 Sep 2021 17:12:59 +0200 Subject: [PATCH 010/148] Trim some useless whitespaces Signed-off-by: Federico Di Pierro Co-authored-by: Leonardo Grasso Co-authored-by: Leonardo Grasso --- driver/bpf/filler_helpers.h | 2 -- 1 file changed, 2 deletions(-) diff --git a/driver/bpf/filler_helpers.h b/driver/bpf/filler_helpers.h index 8f440ac3e..348e461f3 100644 --- a/driver/bpf/filler_helpers.h +++ b/driver/bpf/filler_helpers.h @@ -724,7 +724,6 @@ static __always_inline int __bpf_val_to_ring(struct filler_data *data, curoff_bounded = data->state->tail_ctx.curoff & SCRATCH_SIZE_HALF; if (data->state->tail_ctx.curoff > SCRATCH_SIZE_HALF) return PPM_FAILURE_BUFFER_FULL; - if (dyn_idx != (u8)-1) { *((u8 *)&data->buf[curoff_bounded]) = dyn_idx; len_dyn = sizeof(u8); @@ -874,7 +873,6 @@ static __always_inline int __bpf_val_to_ring(struct filler_data *data, return PPM_FAILURE_BUG; } } - if (len_dyn + len > PPM_MAX_ARG_SIZE) return PPM_FAILURE_BUFFER_FULL; From bb0a00aec97dd438814eff8954ff7c5e644a2cbe Mon Sep 17 00:00:00 2001 From: Jason Dellaluce Date: Fri, 17 Sep 2021 08:05:20 +0000 Subject: [PATCH 011/148] fix(driver/bpf): enhance support to older kernels Signed-off-by: Jason Dellaluce --- driver/bpf/fillers.h | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/driver/bpf/fillers.h b/driver/bpf/fillers.h index 4fe0665c9..2bf292523 100644 --- a/driver/bpf/fillers.h +++ b/driver/bpf/fillers.h @@ -435,6 +435,7 @@ static __always_inline int bpf_parse_readv_writev_bufs(struct filler_data *data, if (flags & PRB_FLAG_PUSH_DATA) { if (size > 0) { unsigned long off = _READ(data->state->tail_ctx.curoff); + unsigned long off_bounded; unsigned long remaining = size; int j; @@ -445,6 +446,7 @@ static __always_inline int bpf_parse_readv_writev_bufs(struct filler_data *data, if (j == iovcnt) break; + off_bounded = off & SCRATCH_SIZE_HALF; if (off > SCRATCH_SIZE_HALF) break; @@ -458,11 +460,11 @@ static __always_inline int bpf_parse_readv_writev_bufs(struct filler_data *data, #ifdef BPF_FORBIDS_ZERO_ACCESS if (to_read) - if (bpf_probe_read(&data->buf[off & SCRATCH_SIZE_HALF], + if (bpf_probe_read(&data->buf[off_bounded], ((to_read - 1) & SCRATCH_SIZE_HALF) + 1, iov[j].iov_base)) #else - if (bpf_probe_read(&data->buf[off & SCRATCH_SIZE_HALF], + if (bpf_probe_read(&data->buf[off_bounded], to_read & SCRATCH_SIZE_HALF, iov[j].iov_base)) #endif From 4d9c85eca2e6c514748525c7c424f303734d45b4 Mon Sep 17 00:00:00 2001 From: lucklypse Date: Wed, 4 Aug 2021 12:08:04 +0000 Subject: [PATCH 012/148] libsinsp: fix formatter for bytebuf types Signed-off-by: lucklypse --- userspace/libsinsp/filter.cpp | 8 ++++---- userspace/libsinsp/filterchecks.cpp | 2 +- userspace/libsinsp/filterchecks.h | 2 +- 3 files changed, 6 insertions(+), 6 deletions(-) diff --git a/userspace/libsinsp/filter.cpp b/userspace/libsinsp/filter.cpp index e3f453ba0..f348655b8 100644 --- a/userspace/libsinsp/filter.cpp +++ b/userspace/libsinsp/filter.cpp @@ -1055,9 +1055,9 @@ char* sinsp_filter_check::rawval_to_string(uint8_t* rawval, { ASSERT(len < 1024 * 1024); - if(len >= filter_value().size()) + if(len >= filter_value()->size()) { - filter_value().resize(len + 1); + filter_value()->resize(len + 1); } memcpy(filter_value_p(), rawval, len); @@ -1209,7 +1209,7 @@ void sinsp_filter_check::add_filter_value(const char* str, uint32_t len, uint32_ m_val_storages.push_back(vector(256)); } - parsed_len = parse_filter_value(str, len, filter_value_p(i), filter_value(i).size()); + parsed_len = parse_filter_value(str, len, filter_value_p(i), filter_value(i)->size()); // XXX/mstemm this doesn't work if someone called // add_filter_value more than once for a given index. @@ -1287,7 +1287,7 @@ bool sinsp_filter_check::flt_compare(cmpop op, ppm_param_type type, void* operan operand1, filter_value_p(i), op1_len, - filter_value(i).size())) + filter_value(i)->size())) { return true; } diff --git a/userspace/libsinsp/filterchecks.cpp b/userspace/libsinsp/filterchecks.cpp index 919c9ab08..868b77d9d 100644 --- a/userspace/libsinsp/filterchecks.cpp +++ b/userspace/libsinsp/filterchecks.cpp @@ -3082,7 +3082,7 @@ size_t sinsp_filter_check_event::parse_filter_value(const char* str, uint32_t le if(m_field_id == sinsp_filter_check_event::TYPE_ARGRAW) { ASSERT(m_arginfo != NULL); - parsed_len = sinsp_filter_value_parser::string_to_rawval(str, len, filter_value_p(), filter_value().size(), m_arginfo->type); + parsed_len = sinsp_filter_value_parser::string_to_rawval(str, len, filter_value_p(), filter_value()->size(), m_arginfo->type); } else { diff --git a/userspace/libsinsp/filterchecks.h b/userspace/libsinsp/filterchecks.h index ef2c79a3e..af424c386 100644 --- a/userspace/libsinsp/filterchecks.h +++ b/userspace/libsinsp/filterchecks.h @@ -173,7 +173,7 @@ class sinsp_filter_check : public gen_event_filter_check char m_getpropertystr_storage[1024]; vector> m_val_storages; inline uint8_t* filter_value_p(uint16_t i = 0) { return &m_val_storages[i][0]; } - inline vector filter_value(uint16_t i = 0) { return m_val_storages[i]; } + inline vector* filter_value(uint16_t i = 0) { return &m_val_storages[i]; } unordered_set Date: Wed, 4 Aug 2021 16:05:54 +0000 Subject: [PATCH 013/148] cleanup(libsinsp): make user agent string parametrized Signed-off-by: lucklypse --- cmake/modules/libsinsp.cmake | 4 ++++ userspace/libsinsp/mesos_http.cpp | 2 +- userspace/libsinsp/sinsp.h | 8 ++++++++ userspace/libsinsp/socket_handler.h | 2 +- 4 files changed, 14 insertions(+), 2 deletions(-) diff --git a/cmake/modules/libsinsp.cmake b/cmake/modules/libsinsp.cmake index 9b184c5ce..c126359d9 100644 --- a/cmake/modules/libsinsp.cmake +++ b/cmake/modules/libsinsp.cmake @@ -9,6 +9,10 @@ option(USE_BUNDLED_DEPS "Enable bundled dependencies instead of using the system option(WITH_CHISEL "Include chisel implementation" OFF) +if(DEFINED LIBSINSP_USER_AGENT) + add_definitions(-DLIBSINSP_USER_AGENT="${LIBSINSP_USER_AGENT}") +endif() + include(ExternalProject) include(libscap) if(NOT WIN32) diff --git a/userspace/libsinsp/mesos_http.cpp b/userspace/libsinsp/mesos_http.cpp index b296dd709..ad40f1242 100644 --- a/userspace/libsinsp/mesos_http.cpp +++ b/userspace/libsinsp/mesos_http.cpp @@ -391,7 +391,7 @@ std::string mesos_http::make_request(uri url, curl_version_info_data* curl_versi { request << '?' << query; } - request << " HTTP/1.1\r\nConnection: Keep-Alive\r\nUser-Agent: sysdig"; + request << " HTTP/1.1\r\nConnection: Keep-Alive\r\nUser-Agent: " LIBSINSP_USER_AGENT; if(curl_version && curl_version->version) { request << " (curl " << curl_version->version << ')'; diff --git a/userspace/libsinsp/sinsp.h b/userspace/libsinsp/sinsp.h index 559fb42a6..f54e5bb66 100644 --- a/userspace/libsinsp/sinsp.h +++ b/userspace/libsinsp/sinsp.h @@ -156,6 +156,14 @@ class metadata_download_params uint32_t m_data_watch_freq_sec = METADATA_DATA_WATCH_FREQ_SEC; }; +/*! + \brief The user agent string to use for any libsinsp connection, can be changed at compile time +*/ + +#if !defined(LIBSINSP_USER_AGENT) +#define LIBSINSP_USER_AGENT "falcosecurity-libs" +#endif // LIBSINSP_USER_AGENT + /*! \brief The default way an event is converted to string by the library */ diff --git a/userspace/libsinsp/socket_handler.h b/userspace/libsinsp/socket_handler.h index 08e429016..df3ab51be 100644 --- a/userspace/libsinsp/socket_handler.h +++ b/userspace/libsinsp/socket_handler.h @@ -181,7 +181,7 @@ class socket_data_handler { request << '?' << query; } - request << " HTTP/" << http_version << "\r\n" << m_keep_alive << "User-Agent: sysdig\r\n"; + request << " HTTP/" << http_version << "\r\n" << m_keep_alive << "User-Agent: " LIBSINSP_USER_AGENT "\r\n"; if(!host_and_port.empty()) { request << "Host: " << host_and_port << "\r\n"; From 3aa7a83bf7b9e6229a3824e3fd1f4452d1e95cb4 Mon Sep 17 00:00:00 2001 From: Michele Zuccala Date: Wed, 29 Sep 2021 11:57:10 +0200 Subject: [PATCH 014/148] fix(cmake): downgrade openssl to 1.0.2u Co-authored-by: Mark Stemm Co-authored-by: Leonardo Grasso Co-authored-by: lucklypse Co-authored-by: Federico Di Pierro Co-authored-by: Jason Dellaluce Signed-off-by: Michele Zuccala --- cmake/modules/openssl.cmake | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/cmake/modules/openssl.cmake b/cmake/modules/openssl.cmake index 1ed471a44..bc9955a99 100644 --- a/cmake/modules/openssl.cmake +++ b/cmake/modules/openssl.cmake @@ -20,8 +20,8 @@ else() ExternalProject_Add(openssl PREFIX "${PROJECT_BINARY_DIR}/openssl-prefix" - URL "https://github.com/openssl/openssl/archive/OpenSSL_1_1_1k.tar.gz" - URL_HASH "SHA256=b92f9d3d12043c02860e5e602e50a73ed21a69947bcc74d391f41148e9f6aa95" + URL "https://github.com/openssl/openssl/archive/OpenSSL_1_0_2u.tar.gz" + URL_HASH "SHA256=82fa58e3f273c53128c6fe7e3635ec8cda1319a10ce1ad50a987c3df0deeef05" CONFIGURE_COMMAND ./config no-shared --prefix=${OPENSSL_INSTALL_DIR} BUILD_COMMAND ${CMD_MAKE} BUILD_IN_SOURCE 1 From 01f2e2a2fb6ba2abdda70f6d2a446b0c940a3c0c Mon Sep 17 00:00:00 2001 From: Mark Stemm Date: Wed, 25 Aug 2021 13:55:32 -0700 Subject: [PATCH 015/148] Add ability to return event types used by a filter Add the ability to return the event types used by a filter. For example, if a filter was "evt.type=open and fd.name=/tmp/foo", the event types would be PPME_SYSCALL_OPEN*_{E,X}. By default, an empty set is returned, meaning no specific events are used. Event types PPME_GENERIC_{E,X} are not included and it's assumed the code using this will handle those event types directly. This is used in programs like falco to provide a quick external test against an event to see if it makes sense to evaluate the filter at all. This can speed up event processing when falco has a large number of loaded rules. Prior to this change, this was handled solely in falco's lua code for loading rules. Moving responsibility to the filter significantly simplifies the falco side of rule loading. In the base classes, new methods gen_event_filter_check::evttypes/possible_evttypes return a set of event types. The base class implementation just returns a single event type "1". gen_event_filter_expression::evttypes() does all the work of iterating over the filterchecks that make up an expression and combining sets of event types. possible_evttypes is used for "not" operators, which invert a set of event types to include everything outside the set. The sinsp "base" class sinsp_filter_check just returns all event types from 2 to PPM_EVENT_MAX. The only actual implementation of evttypes() that does something is in sinsp_filter_check_event for the field "evt.type". The method handles =, in, and != as comparison operators. Also add a unit test that compiles various filters and double-checks the resulting set of event types. Signed-off-by: Mark Stemm --- userspace/libsinsp/filter.cpp | 24 ++ userspace/libsinsp/filterchecks.cpp | 67 ++++ userspace/libsinsp/filterchecks.h | 9 + userspace/libsinsp/gen_filter.cpp | 113 +++++++ userspace/libsinsp/gen_filter.h | 37 +++ userspace/libsinsp/test/CMakeLists.txt | 1 + userspace/libsinsp/test/evttype_filter.ut.cpp | 285 ++++++++++++++++++ 7 files changed, 536 insertions(+) create mode 100644 userspace/libsinsp/test/evttype_filter.ut.cpp diff --git a/userspace/libsinsp/filter.cpp b/userspace/libsinsp/filter.cpp index f348655b8..6b1a3d71f 100644 --- a/userspace/libsinsp/filter.cpp +++ b/userspace/libsinsp/filter.cpp @@ -651,6 +651,18 @@ sinsp_filter_check::sinsp_filter_check() m_val_storages = vector> (1, vector(256)); m_val_storages_min_size = (numeric_limits::max)(); m_val_storages_max_size = (numeric_limits::min)(); + + // Do this once + if(s_all_event_types.size() == 0) + { + // + // Fill in from 2 to PPM_EVENT_MAX-1. 0 and 1 are excluded as + // those are PPM_GENERIC_E/PPME_GENERIC_X. + for(uint16_t i = 2; i < PPM_EVENT_MAX; i++) + { + s_all_event_types.insert(i); + } + } } void sinsp_filter_check::set_inspector(sinsp* inspector) @@ -1365,6 +1377,18 @@ uint8_t* sinsp_filter_check::extract_cached(sinsp_evt *evt, OUT uint32_t* len, b } } +const std::set &sinsp_filter_check::evttypes() +{ + // By default a check should be considered for all event types + // so use the default. + return s_all_event_types; +} + +const std::set &sinsp_filter_check::possible_evttypes() +{ + return s_all_event_types; +} + bool sinsp_filter_check::compare(gen_event *evt) { if(m_eval_cache_entry != NULL) diff --git a/userspace/libsinsp/filterchecks.cpp b/userspace/libsinsp/filterchecks.cpp index 868b77d9d..ba053529e 100644 --- a/userspace/libsinsp/filterchecks.cpp +++ b/userspace/libsinsp/filterchecks.cpp @@ -4524,6 +4524,73 @@ uint8_t* sinsp_filter_check_event::extract(sinsp_evt *evt, OUT uint32_t* len, bo return NULL; } +const std::set &sinsp_filter_check_event::evttypes() +{ + bool should_match = true; + + if(m_field_id == TYPE_TYPE) + { + // The only meaningful comparison operators for this + // filter check should be direct comparisons and not + // LT/GE/CONTAINS/etc. + if(!(m_cmpop == CO_EQ || m_cmpop == CO_NE || m_cmpop == CO_IN)) + { + m_event_types = possible_evttypes(); + return m_event_types; + } + + // If the comparison operator is !=, we need to invert + // the set. "not" is handled in + // gen_event_filter_expression::evttypes(). + if(m_cmpop == CO_NE) + { + should_match = false; + } + } + else + { + // Should run for all event types + m_event_types = possible_evttypes(); + return m_event_types; + } + + // If here, we know we can find a more specific set of event types. + m_event_types.clear(); + + sinsp_evttables* einfo = m_inspector->get_event_info_tables(); + const struct ppm_event_info* etable = einfo->m_event_info; + + // This skips PPME_EVENT_GENERIC_{E,X} as the logical + // operators/inverse don't work for these values. + for(uint32_t i = 2; i < PPM_EVENT_MAX; i++) + { + // The values are held as strings, so we need to + // convert them back to numbers. + bool found = false; + for (uint16_t j=0; j < m_val_storages.size(); j++) + { + std::string evttype_str((char *) filter_value_p(j)); + + if(etable[i].name == evttype_str) + { + found = true; + break; + } + } + + // We add to m_event_types if: + // - was found and should_match == true, or + // - was not found and should_match == false + // so found == should_match + if(found == should_match) + { + m_event_types.insert(i); + } + } + + return m_event_types; +} + bool sinsp_filter_check_event::compare(sinsp_evt *evt) { bool res; diff --git a/userspace/libsinsp/filterchecks.h b/userspace/libsinsp/filterchecks.h index af424c386..0e67bcfc0 100644 --- a/userspace/libsinsp/filterchecks.h +++ b/userspace/libsinsp/filterchecks.h @@ -136,6 +136,9 @@ class sinsp_filter_check : public gen_event_filter_check return Json::nullValue; } + virtual const std::set &evttypes() override; + const std::set &possible_evttypes() override; + // // Compare the field with the constant value obtained from parse_filter_value() // @@ -193,6 +196,8 @@ class sinsp_filter_check : public gen_event_filter_check private: void set_inspector(sinsp* inspector); + std::set s_all_event_types; + friend class sinsp_filter_check_list; friend class sinsp_filter_optimizer; friend class chk_compare_helper; @@ -482,6 +487,8 @@ class sinsp_filter_check_event : public sinsp_filter_check Json::Value extract_as_js(sinsp_evt *evt, OUT uint32_t* len); bool compare(sinsp_evt *evt); + const std::set &evttypes() override; + uint64_t m_u64val; uint64_t m_tsdelta; uint32_t m_u32val; @@ -510,6 +517,8 @@ class sinsp_filter_check_event : public sinsp_filter_check uint32_t m_storage_size; const char* m_cargname; sinsp_filter_check_reference* m_converter; + + std::set m_event_types; }; // diff --git a/userspace/libsinsp/gen_filter.cpp b/userspace/libsinsp/gen_filter.cpp index b2a13acdb..add57dae7 100644 --- a/userspace/libsinsp/gen_filter.cpp +++ b/userspace/libsinsp/gen_filter.cpp @@ -15,11 +15,14 @@ along with Falco. If not, see . */ #include +#include #include "stdint.h" #include "gen_filter.h" #include "sinsp.h" #include "sinsp_int.h" +std::set gen_event_filter_check::s_default_evttypes{1}; + gen_event::gen_event() { } @@ -58,6 +61,16 @@ int32_t gen_event_filter_check::get_check_id() return m_check_id; } +const std::set &gen_event_filter_check::evttypes() +{ + return s_default_evttypes; +} + +const std::set &gen_event_filter_check::possible_evttypes() +{ + return s_default_evttypes; +} + /////////////////////////////////////////////////////////////////////////////// // gen_event_filter_expression implementation /////////////////////////////////////////////////////////////////////////////// @@ -199,6 +212,101 @@ int32_t gen_event_filter_expression::get_expr_boolop() return b0; } +std::set gen_event_filter_expression::inverse(const std::set &evttypes) +{ + std::set ret; + + // The inverse of "all events" is still "all events". This + // ensures that when no specific set of event types are named + // in the filter that the filter still runs for all event + // types. + if(evttypes == m_expr_possible_evttypes) + { + ret = evttypes; + return ret; + } + + std::set_difference(m_expr_possible_evttypes.begin(), m_expr_possible_evttypes.end(), + evttypes.begin(), evttypes.end(), + std::inserter(ret, ret.begin())); + + return ret; +} + +void gen_event_filter_expression::combine_evttypes(boolop op, + const std::set &chk_evttypes) +{ + switch(op) + { + case BO_NONE: + // Overwrite with contents of set + // Should only occur for the first check in a list + m_expr_event_types = chk_evttypes; + break; + case BO_NOT: + m_expr_event_types = inverse(chk_evttypes); + break; + case BO_ORNOT: + combine_evttypes(BO_OR, inverse(chk_evttypes)); + break; + case BO_ANDNOT: + combine_evttypes(BO_AND, inverse(chk_evttypes)); + break; + case BO_OR: + // Merge the event types from the + // other set into this one. + m_expr_event_types.insert(chk_evttypes.begin(), chk_evttypes.end()); + break; + case BO_AND: + // Set to the intersection of event types between this + // set and the provided set. + + std::set intersect; + std::set_intersection(m_expr_event_types.begin(), m_expr_event_types.end(), + chk_evttypes.begin(), chk_evttypes.end(), + std::inserter(intersect, intersect.begin())); + m_expr_event_types = intersect; + break; + } +} + +const std::set &gen_event_filter_expression::evttypes() +{ + m_expr_event_types.clear(); + + m_expr_possible_evttypes = possible_evttypes(); + + for(uint32_t i = 0; i < m_checks.size(); i++) + { + gen_event_filter_check *chk = m_checks[i]; + ASSERT(chk != NULL); + + const std::set &chk_evttypes = m_checks[i]->evttypes(); + + combine_evttypes(chk->m_boolop, chk_evttypes); + } + + return m_expr_event_types; +} + +const std::set &gen_event_filter_expression::possible_evttypes() +{ + // Return the set of possible event types from the first filtercheck. + if(m_checks.size() == 0) + { + // Shouldn't happen--every filter expression should have a + // real filtercheck somewhere below it. + ASSERT(false); + m_expr_possible_evttypes = s_default_evttypes; + } + else + { + m_expr_possible_evttypes = m_checks[0]->possible_evttypes(); + } + + return m_expr_possible_evttypes; +} + /////////////////////////////////////////////////////////////////////////////// // sinsp_filter implementation /////////////////////////////////////////////////////////////////////////////// @@ -248,3 +356,8 @@ void gen_event_filter::add_check(gen_event_filter_check* chk) { m_curexpr->add_check((gen_event_filter_check *) chk); } + +std::set gen_event_filter::evttypes() +{ + return m_filter->evttypes(); +} diff --git a/userspace/libsinsp/gen_filter.h b/userspace/libsinsp/gen_filter.h index a9ee2a7a6..c6f296d43 100644 --- a/userspace/libsinsp/gen_filter.h +++ b/userspace/libsinsp/gen_filter.h @@ -16,6 +16,7 @@ along with Falco. If not, see . #pragma once +#include #include /* @@ -115,6 +116,16 @@ class gen_event_filter_check void set_check_id(int32_t id); virtual int32_t get_check_id(); + // Return all event types used by this filtercheck. It's used in + // programs like falco to speed up rule evaluation. + virtual const std::set &evttypes(); + + // Return all possible event types. Used for "not" operators + // where a set of events must be inverted. + virtual const std::set &possible_evttypes(); + + static std::set s_default_evttypes; + private: int32_t m_check_id = 0; @@ -160,8 +171,30 @@ class gen_event_filter_expression : public gen_event_filter_check // int32_t get_expr_boolop(); + // Return all event types used by this expression. It's used in + // programs like falco to speed up rule evaluation. + const std::set &evttypes() override; + + // An expression does not directly have a set of possible + // event types, but it can determine them from the m_checks + // vector. + const std::set &possible_evttypes() override; + gen_event_filter_expression* m_parent; std::vector m_checks; + +private: + + std::set m_expr_event_types; + std::set m_expr_possible_evttypes; + + // Return the "inverse" of the provided set of event types, using the + // provided full possible set of event types as a hint. + std::set inverse(const std::set &evttypes); + + // Given a boolean op and a set of event types from a + // filtercheck in the expression, update m_expr_event_types appropriately. + void combine_evttypes(boolop op, const std::set &evttypes); }; @@ -184,6 +217,10 @@ class gen_event_filter void pop_expression(); void add_check(gen_event_filter_check* chk); + // Return all event types used by this filter. It's used in + // programs like falco to speed up rule evaluation. + std::set evttypes(); + gen_event_filter_expression* m_filter; protected: diff --git a/userspace/libsinsp/test/CMakeLists.txt b/userspace/libsinsp/test/CMakeLists.txt index a8e761735..651d2586c 100644 --- a/userspace/libsinsp/test/CMakeLists.txt +++ b/userspace/libsinsp/test/CMakeLists.txt @@ -27,6 +27,7 @@ include_directories(${LIBSCAP_INCLUDE_DIR}) add_executable(unit-test-libsinsp cgroup_list_counter.ut.cpp sinsp.ut.cpp + evttype_filter.ut.cpp ) target_link_libraries(unit-test-libsinsp diff --git a/userspace/libsinsp/test/evttype_filter.ut.cpp b/userspace/libsinsp/test/evttype_filter.ut.cpp new file mode 100644 index 000000000..7b9027490 --- /dev/null +++ b/userspace/libsinsp/test/evttype_filter.ut.cpp @@ -0,0 +1,285 @@ +/* +Copyright (C) 2021 The Falco Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. + +*/ + +#include +#include +#include + +std::stringstream & operator<<(std::stringstream &out, set s) +{ + out << "[ "; + for(auto &val : s) + { + out << val; + out << " "; + } + out << "]"; + + return out; +} + +class evttype_filter_test : public testing::Test +{ + +protected: + + void SetUp() + { + for(uint32_t i = 2; i < PPM_EVENT_MAX; i++) + { + all_events.insert(i); + + if(openat_only.find(i) == openat_only.end()) + { + not_openat.insert(i); + } + + if(openat_close.find(i) == openat_close.end()) + { + not_openat_close.insert(i); + } + + if (close_only.find(i) == close_only.end()) + { + not_close.insert(i); + } + } + } + + void TearDown() + { + } + + sinsp_filter *compile(const string &fltstr) + { + sinsp_filter_compiler compiler(NULL, fltstr); + + return compiler.compile(); + } + + void compare_evttypes(sinsp_filter *f, std::set &expected) + { + std::set actual = f->evttypes(); + + for(auto &etype : expected) + { + if(actual.find(etype) == actual.end()) + { + FAIL() << "Expected event type " + << etype + << " not found in actual set. " + << "Expected: " << expected << " " + << " Actual: " << actual; + + } + } + + for(auto &etype : actual) + { + if(expected.find(etype) == expected.end()) + { + FAIL() << "Actual evttypes had additional event type " + << etype + << " not found in expected set. " + << "Expected: " << expected << " " + << " Actual: " << actual; + } + } + } + + std::set openat_only{ + PPME_SYSCALL_OPENAT_E, PPME_SYSCALL_OPENAT_X, + PPME_SYSCALL_OPENAT_2_E, PPME_SYSCALL_OPENAT_2_X + }; + + std::set close_only{ + PPME_SYSCALL_CLOSE_E, PPME_SYSCALL_CLOSE_X + }; + + std::set openat_close{ + PPME_SYSCALL_OPENAT_E, PPME_SYSCALL_OPENAT_X, + PPME_SYSCALL_OPENAT_2_E, PPME_SYSCALL_OPENAT_2_X, + PPME_SYSCALL_CLOSE_E, PPME_SYSCALL_CLOSE_X + }; + + std::set not_openat; + std::set not_openat_close; + std::set not_close; + std::set all_events; + std::set no_events; +}; + +TEST_F(evttype_filter_test, evt_type_eq) +{ + sinsp_filter *f = compile("evt.type=openat"); + + compare_evttypes(f, openat_only); +} + +TEST_F(evttype_filter_test, evt_type_in) +{ + sinsp_filter *f = compile("evt.type in (openat, close)"); + + compare_evttypes(f, openat_close); +} + +TEST_F(evttype_filter_test, evt_type_ne) +{ + sinsp_filter *f = compile("evt.type!=openat"); + + compare_evttypes(f, not_openat); +} + +TEST_F(evttype_filter_test, not_evt_type_eq) +{ + sinsp_filter *f = compile("not evt.type=openat"); + + compare_evttypes(f, not_openat); +} + +TEST_F(evttype_filter_test, not_evt_type_in) +{ + sinsp_filter *f = compile("not evt.type in (openat, close)"); + + compare_evttypes(f, not_openat_close); +} + +TEST_F(evttype_filter_test, not_evt_type_ne) +{ + sinsp_filter *f = compile("not evt.type != openat"); + + compare_evttypes(f, openat_only); +} + +TEST_F(evttype_filter_test, evt_type_or) +{ + sinsp_filter *f = compile("evt.type=openat or evt.type=close"); + + compare_evttypes(f, openat_close); +} + +TEST_F(evttype_filter_test, not_evt_type_or) +{ + sinsp_filter *f = compile("evt.type!=openat or evt.type!=close"); + + compare_evttypes(f, all_events); +} + +TEST_F(evttype_filter_test, evt_type_or_ne) +{ + sinsp_filter *f = compile("evt.type=close or evt.type!=openat"); + + compare_evttypes(f, not_openat); +} + +TEST_F(evttype_filter_test, evt_type_and) +{ + sinsp_filter *f = compile("evt.type=close and evt.type=openat"); + + compare_evttypes(f, no_events); +} + +TEST_F(evttype_filter_test, evt_type_and_non_evt_type) +{ + sinsp_filter *f = compile("evt.type=openat and proc.name=nginx"); + + compare_evttypes(f, openat_only); +} + +TEST_F(evttype_filter_test, evt_type_and_non_evt_type_not) +{ + sinsp_filter *f = compile("evt.type=openat and not proc.name=nginx"); + + compare_evttypes(f, openat_only); +} + +TEST_F(evttype_filter_test, evt_type_and_nested) +{ + sinsp_filter *f = compile("evt.type=openat and (proc.name=nginx)"); + + compare_evttypes(f, openat_only); +} + +TEST_F(evttype_filter_test, evt_type_and_nested_multi) +{ + sinsp_filter *f = compile("evt.type=openat and (evt.type=close and proc.name=nginx)"); + + compare_evttypes(f, no_events); +} + +TEST_F(evttype_filter_test, non_evt_type) +{ + sinsp_filter *f = compile("proc.name=nginx"); + + compare_evttypes(f, all_events); +} + +TEST_F(evttype_filter_test, non_evt_type_or) +{ + sinsp_filter *f = compile("evt.type=openat or proc.name=nginx"); + + compare_evttypes(f, all_events); +} + +TEST_F(evttype_filter_test, non_evt_type_or_nested_first) +{ + sinsp_filter *f = compile("(evt.type=openat) or proc.name=nginx"); + + compare_evttypes(f, all_events); +} + +TEST_F(evttype_filter_test, non_evt_type_or_nested_second) +{ + sinsp_filter *f = compile("evt.type=openat or (proc.name=nginx)"); + + compare_evttypes(f, all_events); +} + +TEST_F(evttype_filter_test, non_evt_type_or_nested_multi) +{ + sinsp_filter *f = compile("evt.type=openat or (evt.type=close and proc.name=nginx)"); + + compare_evttypes(f, openat_close); +} + +TEST_F(evttype_filter_test, non_evt_type_or_nested_multi_not) +{ + sinsp_filter *f = compile("evt.type=openat or not (evt.type=close and proc.name=nginx)"); + + compare_evttypes(f, not_close); +} + +TEST_F(evttype_filter_test, non_evt_type_and_nested_multi_not) +{ + sinsp_filter *f = compile("evt.type=openat and not (evt.type=close and proc.name=nginx)"); + + compare_evttypes(f, openat_only); +} + +TEST_F(evttype_filter_test, ne_and_and) +{ + sinsp_filter *f = compile("evt.type!=openat and evt.type!=close"); + + compare_evttypes(f, not_openat_close); +} + +TEST_F(evttype_filter_test, not_not) +{ + sinsp_filter *f = compile("not (not evt.type=openat)"); + + compare_evttypes(f, openat_only); +} From b5308ce965006bfe1b79d55e4838b9357c9ca382 Mon Sep 17 00:00:00 2001 From: Mark Stemm Date: Wed, 25 Aug 2021 14:54:41 -0700 Subject: [PATCH 016/148] Update lua parser to be short-lived for a single filter Clean up the interface for lua_parser/lua_parser_api so it doesn't rely on a single object. Context--the lua_parser object simply holds some intermediate state (e.g. nesting level) and builds up a gen_filter as the lua side traverses the ast of a filter expression. The lua_parser_api class just has static methods that are registered into lua. lua_parser used to be a single object that was reset for each filter. Now, it's an object that is created as a single filter is parsed and deleted afterward. The callbacks always pass the lua_parser object as a first argument and the state/filter in the object is updated. Also, registering the lua callbacks is now done via a static method instead of in the constructor. Signed-off-by: Mark Stemm --- userspace/chisel/lua_parser.cpp | 56 +++++++++-------------------- userspace/chisel/lua_parser.h | 15 ++++---- userspace/chisel/lua_parser_api.cpp | 16 ++++----- 3 files changed, 31 insertions(+), 56 deletions(-) diff --git a/userspace/chisel/lua_parser.cpp b/userspace/chisel/lua_parser.cpp index 66d62841f..f42d47694 100644 --- a/userspace/chisel/lua_parser.cpp +++ b/userspace/chisel/lua_parser.cpp @@ -16,9 +16,8 @@ limitations under the License. */ #include #include -#include "sinsp.h" -#include "filter.h" -#include "sinsp_int.h" +#include +#include "gen_filter.h" #include "lua_parser.h" #include "lua_parser_api.h" @@ -39,50 +38,29 @@ const static struct luaL_reg ll_filter [] = {NULL,NULL} }; -lua_parser::lua_parser(gen_event_filter_factory &factory, lua_State *ls, const char *lua_library_name) - : m_factory(factory) +lua_parser::lua_parser(std::shared_ptr factory) + : m_factory(factory), m_filter(m_factory->new_filter()), + m_last_boolop(BO_NONE), m_have_rel_expr(false), + m_nest_level(0) { - m_filter = NULL; - - m_ls = ls; - reset(); - - // Register our c++ defined functions - luaL_openlib(m_ls, lua_library_name, ll_filter, 0); } -void lua_parser::reset() +lua_parser::~lua_parser() { - m_have_rel_expr = false; - m_last_boolop = BO_NONE; - m_nest_level = 0; - - m_filter = m_factory.new_filter(); } -gen_event_filter* lua_parser::get_filter(bool reset_filter) +void lua_parser::register_callbacks(lua_State *ls, const char *lua_library_name) { - if (m_nest_level != 0) - { - throw sinsp_exception("Error in configured filter: unbalanced nesting"); - } - - gen_event_filter *ret = m_filter; - - if (reset_filter) - { - reset(); - } - - return ret; + // Register our c++ defined functions + luaL_openlib(ls, lua_library_name, ll_filter, 0); } -lua_parser::~lua_parser() -{ - // The lua state is not considered owned by this object, so - // not freeing it. - delete m_filter; - m_filter = NULL; +std::shared_ptr lua_parser::filter() +{ + return m_filter; } - +std::shared_ptr lua_parser::factory() +{ + return m_factory; +} diff --git a/userspace/chisel/lua_parser.h b/userspace/chisel/lua_parser.h index a3afc91da..173e814a2 100644 --- a/userspace/chisel/lua_parser.h +++ b/userspace/chisel/lua_parser.h @@ -24,23 +24,22 @@ typedef struct lua_State lua_State; class lua_parser { public: - lua_parser(gen_event_filter_factory &factory, lua_State *ls, const char *lua_library_name); + lua_parser(std::shared_ptr factory); virtual ~lua_parser(); - gen_event_filter* get_filter(bool reset_filter = false); - private: + std::shared_ptr filter(); + std::shared_ptr factory(); - void reset(); - gen_event_filter_factory &m_factory; + static void register_callbacks(lua_State *ls, const char *lua_library_name); - gen_event_filter* m_filter; + private: + std::shared_ptr m_factory; + std::shared_ptr m_filter; boolop m_last_boolop; bool m_have_rel_expr; int32_t m_nest_level; - lua_State* m_ls; - friend class lua_parser_cbacks; }; diff --git a/userspace/chisel/lua_parser_api.cpp b/userspace/chisel/lua_parser_api.cpp index 1d8dc311e..1c5e00365 100644 --- a/userspace/chisel/lua_parser_api.cpp +++ b/userspace/chisel/lua_parser_api.cpp @@ -14,6 +14,9 @@ See the License for the specific language governing permissions and limitations under the License. */ + +#include + #include "sinsp.h" #include "sinsp_int.h" @@ -131,9 +134,7 @@ int lua_parser_cbacks::nest(lua_State *ls) throw sinsp_exception(err); } - gen_event_filter* filter = parser->m_filter; - - filter->push_expression(parser->m_last_boolop); + parser->filter()->push_expression(parser->m_last_boolop); parser->m_nest_level++; parser->m_last_boolop = BO_NONE; @@ -160,9 +161,7 @@ int lua_parser_cbacks::unnest(lua_State *ls) throw sinsp_exception(err); } - gen_event_filter* filter = parser->m_filter; - - filter->pop_expression(); + parser->filter()->pop_expression(); parser->m_nest_level--; } catch (const std::exception& e) @@ -231,10 +230,9 @@ int lua_parser_cbacks::rel_expr(lua_State *ls) } parser->m_have_rel_expr = true; - gen_event_filter* filter = parser->m_filter; const char* fld = luaL_checkstring(ls, 2); - gen_event_filter_check *chk = parser->m_factory.new_filtercheck(fld); + gen_event_filter_check *chk = parser->factory()->new_filtercheck(fld); if(chk == NULL) { string err = "filter_check called with nonexistent field " + string(fld); @@ -297,7 +295,7 @@ int lua_parser_cbacks::rel_expr(lua_State *ls) chk->set_check_id(rule_index); } - filter->add_check(chk); + parser->filter()->add_check(chk); } catch (const std::exception& e) From 59a81453d874bf07eeaedf66f5bf0381d8eef110 Mon Sep 17 00:00:00 2001 From: Mark Stemm Date: Wed, 25 Aug 2021 14:18:15 -0700 Subject: [PATCH 017/148] Add ability to return all fields exported by a factory Add the ability to return all fields exported by a factory. This is important for programs like falco that need to validate rule filter expressions for various event sources, as well as print out sets of supported fields. Previously, falco did direct calls to sinsp::get_filtercheck_fields_info but we're trying to standardize everything to work through factories, to make it easier to support new event sources. This PR supports that work. Signed-off-by: Mark Stemm --- userspace/libsinsp/filter.cpp | 45 +++++++++++++++++++++++++++++++++ userspace/libsinsp/filter.h | 2 ++ userspace/libsinsp/gen_filter.h | 32 ++++++++++++++++++++++- 3 files changed, 78 insertions(+), 1 deletion(-) diff --git a/userspace/libsinsp/filter.cpp b/userspace/libsinsp/filter.cpp index 6b1a3d71f..8ae9aa207 100644 --- a/userspace/libsinsp/filter.cpp +++ b/userspace/libsinsp/filter.cpp @@ -2442,5 +2442,50 @@ gen_event_filter_check *sinsp_filter_factory::new_filtercheck(const char *fldnam true); } +std::list sinsp_filter_factory::get_fields() +{ + std::list ret; + + vector fc_plugins; + sinsp::get_filtercheck_fields_info(&fc_plugins); + + for(auto &fci : fc_plugins) + { + if(fci->m_flags & filter_check_info::FL_HIDDEN) + { + continue; + } + + gen_event_filter_factory::filter_fieldclass_info cinfo; + cinfo.name = fci->m_name; + cinfo.desc = ""; + cinfo.class_info = ""; + + for(int32_t k = 0; k < fci->m_nfields; k++) + { + const filtercheck_field_info* fld = &fci->m_fields[k]; + + // If a field can't be used to filter events, + // or if a field is only used for stuff like + // chisels to organize events, we don't want + // to print it and don't return it here. + if(fld->m_flags & EPF_TABLE_ONLY || + fld->m_flags & EPF_PRINT_ONLY) + { + continue; + } + + gen_event_filter_factory::filter_field_info info; + info.name = fld->m_name; + info.desc = fld->m_description; + + cinfo.fields.emplace_back(std::move(info)); + } + + ret.emplace_back(std::move(cinfo)); + } + + return ret; +} #endif // HAS_FILTERING diff --git a/userspace/libsinsp/filter.h b/userspace/libsinsp/filter.h index 2edff7142..33acb5749 100644 --- a/userspace/libsinsp/filter.h +++ b/userspace/libsinsp/filter.h @@ -213,6 +213,8 @@ class sinsp_filter_factory : public gen_event_filter_factory gen_event_filter_check *new_filtercheck(const char *fldname); + std::list get_fields() override; + protected: sinsp *m_inspector; }; diff --git a/userspace/libsinsp/gen_filter.h b/userspace/libsinsp/gen_filter.h index c6f296d43..8aa35c64c 100644 --- a/userspace/libsinsp/gen_filter.h +++ b/userspace/libsinsp/gen_filter.h @@ -17,6 +17,8 @@ along with Falco. If not, see . #pragma once #include +#include +#include #include /* @@ -166,7 +168,7 @@ class gen_event_filter_expression : public gen_event_filter_check // // An expression is consistent if all its checks are of the same type (or/and). // - // This method returns the expression operator (BO_AND/BO_OR/BO_NONE) if the + // This method returns the expression operator (BO_AND/BO_OR/BO_NONE) if the // expression is consistent. It returns -1 if the expression is not consistent. // int32_t get_expr_boolop(); @@ -234,6 +236,31 @@ class gen_event_filter_factory { public: + // A struct describing a single filtercheck field ("ka.user") + struct filter_field_info + { + // The name of the field + std::string name; + + // A description of the field + std::string desc; + }; + + // A struct describing a group of filtercheck fields ("ka") + struct filter_fieldclass_info + { + // The name of the group of fields + std::string name; + + // A short description for the fields + std::string desc; + + // Additional information about proper use of the fields + std::string class_info; + + std::list fields; + }; + gen_event_filter_factory() {}; virtual ~gen_event_filter_factory() {}; @@ -242,4 +269,7 @@ class gen_event_filter_factory // Create a new filtercheck virtual gen_event_filter_check *new_filtercheck(const char *fldname) = 0; + + // Return the set of fields supported by this factory + virtual std::list get_fields() = 0; }; From a5ea3673684f33316d6e7897777982e9b042cbc8 Mon Sep 17 00:00:00 2001 From: Mark Stemm Date: Wed, 25 Aug 2021 15:11:39 -0700 Subject: [PATCH 018/148] Add notion of generic formatters/formatter factories Add the notion of "generic" event formatters and formatter factories that can create a formatter for a given format string: - gen_filter.h defines two new classes: gen_event_formatter provides the interface to format events: - set_format(): set the output format and format string - tostring(): resolve the format with info from a gen_event, populating a resolved string. - tostring_withformat(): like tostring() but with a one-off output format. - get_field_values(): return all templated field names and values from the configured format string. - get_output_format(): get the current output format. - gen_event_formatter_factory performs a similar role as the existing sinsp_evt_formatter_cache, in that it maintains a cache of previously used format strings/formatters, to avoid the overhead of creating formatters. It simply returns a formatter, though, rather than duplicating the format methods like sinsp_evt_formatter_cache does. This can be used in programs like falco to format general purpose events without having a direct connection to an inspector/filterchecks/etc. - The eventformatter changes simply add gen_event_formatter as a parent class and implements the interfaces. To aid in backwards compatibility with other parts of libsinsp, this only adds new methods as needed to conform to the gen_event_formatter interface. In some cases, the new methods just call existing methods that did the same thing. Signed-off-by: Mark Stemm --- userspace/libsinsp/eventformatter.cpp | 119 +++++++++++++++++++++----- userspace/libsinsp/eventformatter.h | 41 ++++++++- userspace/libsinsp/gen_filter.cpp | 15 ++++ userspace/libsinsp/gen_filter.h | 44 ++++++++++ 4 files changed, 194 insertions(+), 25 deletions(-) diff --git a/userspace/libsinsp/eventformatter.cpp b/userspace/libsinsp/eventformatter.cpp index ded0071d9..bd702a2ad 100644 --- a/userspace/libsinsp/eventformatter.cpp +++ b/userspace/libsinsp/eventformatter.cpp @@ -27,10 +27,27 @@ limitations under the License. #ifdef HAS_FILTERING extern sinsp_filter_check_list g_filterlist; +sinsp_evt_formatter::sinsp_evt_formatter(sinsp* inspector) + : m_inspector(inspector) +{ +} + sinsp_evt_formatter::sinsp_evt_formatter(sinsp* inspector, const string& fmt) { m_inspector = inspector; - set_format(fmt); + + gen_event_formatter::output_format of = gen_event_formatter::OF_NORMAL; + + if(m_inspector->get_buffer_format() == sinsp_evt::PF_JSON + || m_inspector->get_buffer_format() == sinsp_evt::PF_JSONEOLS + || m_inspector->get_buffer_format() == sinsp_evt::PF_JSONHEX + || m_inspector->get_buffer_format() == sinsp_evt::PF_JSONHEXASCII + || m_inspector->get_buffer_format() == sinsp_evt::PF_JSONBASE64) + { + of = gen_event_formatter::OF_JSON; + } + + set_format(of, fmt); } sinsp_evt_formatter::~sinsp_evt_formatter() @@ -43,12 +60,14 @@ sinsp_evt_formatter::~sinsp_evt_formatter() } } -void sinsp_evt_formatter::set_format(const string& fmt) +void sinsp_evt_formatter::set_format(gen_event_formatter::output_format of, const string& fmt) { uint32_t j; uint32_t last_nontoken_str_start = 0; string lfmt(fmt); + m_output_format = of; + if(lfmt == "") { throw sinsp_exception("empty formatting token"); @@ -200,25 +219,33 @@ bool sinsp_evt_formatter::resolve_tokens(sinsp_evt *evt, map& val return retval; } +bool sinsp_evt_formatter::get_field_values(gen_event *gevt, std::map &fields) +{ + sinsp_evt *evt = static_cast(gevt); -bool sinsp_evt_formatter::tostring(sinsp_evt* evt, OUT string* res) + return resolve_tokens(evt, fields); +} + +gen_event_formatter::output_format sinsp_evt_formatter::get_output_format() +{ + return m_output_format; +} + +bool sinsp_evt_formatter::tostring_withformat(gen_event* gevt, std::string &output, gen_event_formatter::output_format of) { bool retval = true; const filtercheck_field_info* fi; + sinsp_evt *evt = static_cast(gevt); + uint32_t j = 0; - vector::iterator it; - res->clear(); + output.clear(); ASSERT(m_tokenlens.size() == m_tokens.size()); for(j = 0; j < m_tokens.size(); j++) { - if(m_inspector->get_buffer_format() == sinsp_evt::PF_JSON - || m_inspector->get_buffer_format() == sinsp_evt::PF_JSONEOLS - || m_inspector->get_buffer_format() == sinsp_evt::PF_JSONHEX - || m_inspector->get_buffer_format() == sinsp_evt::PF_JSONHEXASCII - || m_inspector->get_buffer_format() == sinsp_evt::PF_JSONBASE64) + if(of == OF_JSON) { Json::Value json_value = m_tokens[j].second->tojson(evt); @@ -268,35 +295,41 @@ bool sinsp_evt_formatter::tostring(sinsp_evt* evt, OUT string* res) { string sstr(str); sstr.resize(tks, ' '); - (*res) += sstr; + output += sstr; } else { - (*res) += str; + output += str; } } } - if(m_inspector->get_buffer_format() == sinsp_evt::PF_JSON - || m_inspector->get_buffer_format() == sinsp_evt::PF_JSONEOLS - || m_inspector->get_buffer_format() == sinsp_evt::PF_JSONHEX - || m_inspector->get_buffer_format() == sinsp_evt::PF_JSONHEXASCII - || m_inspector->get_buffer_format() == sinsp_evt::PF_JSONBASE64) + if(of == OF_JSON) { - (*res) = m_writer.write(m_root); - (*res) = res->substr(0, res->size() - 1); + output = m_writer.write(m_root); + output = output.substr(0, output.size() - 1); } return retval; } +bool sinsp_evt_formatter::tostring(gen_event* gevt, std::string &output) +{ + return tostring_withformat(gevt, output, m_output_format); +} + +bool sinsp_evt_formatter::tostring(sinsp_evt* evt, OUT string* res) +{ + return tostring_withformat(evt, *res, m_output_format); +} + #else // HAS_FILTERING -sinsp_evt_formatter::sinsp_evt_formatter(sinsp* inspector, const string& fmt) +sinsp_evt_formatter::sinsp_evt_formatter(sinsp* inspector) { } -void sinsp_evt_formatter::set_format(const string& fmt) +void sinsp_evt_formatter::set_format(gen_event_formatter::output_format of, const std::string &format) = 0; { throw sinsp_exception("sinsp_evt_formatter unavailable because it was not compiled in the library"); } @@ -306,7 +339,12 @@ bool sinsp_evt_formatter::resolve_tokens(sinsp_evt *evt, map& val throw sinsp_exception("sinsp_evt_formatter unavailable because it was not compiled in the library"); } -bool sinsp_evt_formatter::tostring(sinsp_evt* evt, OUT string* res) +bool sinsp_evt_formatter::tostring(gen_event* gevt, std::string &output) +{ + throw sinsp_exception("sinsp_evt_formatter unavailable because it was not compiled in the library"); +} + +bool sinsp_evt_formatter::tostring_withformat(gen_event* gevt, std::string &output, gen_event_formatter::output_format of) { throw sinsp_exception("sinsp_evt_formatter unavailable because it was not compiled in the library"); } @@ -342,5 +380,40 @@ bool sinsp_evt_formatter_cache::resolve_tokens(sinsp_evt *evt, string &format, m bool sinsp_evt_formatter_cache::tostring(sinsp_evt *evt, string &format, OUT string *res) { - return get_cached_formatter(format)->tostring(evt, res); + return get_cached_formatter(format)->tostring(evt, *res); +} + +sinsp_evt_formatter_factory::sinsp_evt_formatter_factory(sinsp *inspector) + : m_inspector(inspector), m_output_format(gen_event_formatter::OF_NORMAL) +{ +} + +sinsp_evt_formatter_factory::~sinsp_evt_formatter_factory() +{ +} + +void sinsp_evt_formatter_factory::set_output_format(gen_event_formatter::output_format of) +{ + m_formatters.clear(); + + m_output_format = of; +} + +std::shared_ptr sinsp_evt_formatter_factory::create_formatter(const std::string &format) +{ + auto it = m_formatters.find(format); + + if (it != m_formatters.end()) + { + return it->second; + } + + std::shared_ptr ret; + + ret.reset(new sinsp_evt_formatter(m_inspector)); + + ret->set_format(m_output_format, format); + m_formatters[format] = ret; + + return ret; } diff --git a/userspace/libsinsp/eventformatter.h b/userspace/libsinsp/eventformatter.h index 0aa227673..3294bf370 100644 --- a/userspace/libsinsp/eventformatter.h +++ b/userspace/libsinsp/eventformatter.h @@ -16,6 +16,9 @@ limitations under the License. */ #pragma once +#include +#include +#include #include class sinsp_filter_check; @@ -29,7 +32,7 @@ class sinsp_filter_check; This class can be used to format an event into a string, based on an arbitrary format. */ -class SINSP_PUBLIC sinsp_evt_formatter +class SINSP_PUBLIC sinsp_evt_formatter : public gen_event_formatter { public: /*! @@ -41,8 +44,12 @@ class SINSP_PUBLIC sinsp_evt_formatter as the one of the sysdig '-p' command line flag, so refer to the sysdig manual for details. */ + sinsp_evt_formatter(sinsp* inspector); + sinsp_evt_formatter(sinsp* inspector, const string& fmt); + void set_format(gen_event_formatter::output_format of, const string& fmt) override; + ~sinsp_evt_formatter(); /*! @@ -57,6 +64,12 @@ class SINSP_PUBLIC sinsp_evt_formatter */ bool resolve_tokens(sinsp_evt *evt, map& values); + // For compatibility with gen_event_filter_factory + // interface. It just calls resolve_tokens(). + bool get_field_values(gen_event *evt, std::map &fields) override; + + gen_event_formatter::output_format get_output_format() override; + /*! \brief Fills res with the string rendering of the event. @@ -68,6 +81,11 @@ class SINSP_PUBLIC sinsp_evt_formatter */ bool tostring(sinsp_evt* evt, OUT string* res); + // For compatibility with gen_event_formatter + bool tostring(gen_event* evt, std::string &output) override; + + bool tostring_withformat(gen_event* evt, std::string &output, gen_event_formatter::output_format of) override; + /*! \brief Fills res with end of capture string rendering of the event. \param res Pointer to the string that will be filled with the result. @@ -78,7 +96,7 @@ class SINSP_PUBLIC sinsp_evt_formatter bool on_capture_end(OUT string* res); private: - void set_format(const string& fmt); + gen_event_formatter::output_format m_output_format; // vector of (full string of the token, filtercheck) pairs // e.g. ("proc.aname[2], ptr to sinsp_filter_check_thread) @@ -123,3 +141,22 @@ class SINSP_PUBLIC sinsp_evt_formatter_cache sinsp *m_inspector; }; /*@}*/ + +class sinsp_evt_formatter_factory : public gen_event_formatter_factory +{ +public: + sinsp_evt_formatter_factory(sinsp *inspector); + virtual ~sinsp_evt_formatter_factory(); + + void set_output_format(gen_event_formatter::output_format of) override; + + std::shared_ptr create_formatter(const std::string &format) override; + +protected: + + // Maps from output string to formatter + std::map> m_formatters; + + sinsp *m_inspector; + gen_event_formatter::output_format m_output_format; +}; diff --git a/userspace/libsinsp/gen_filter.cpp b/userspace/libsinsp/gen_filter.cpp index add57dae7..4298dad23 100644 --- a/userspace/libsinsp/gen_filter.cpp +++ b/userspace/libsinsp/gen_filter.cpp @@ -361,3 +361,18 @@ std::set gen_event_filter::evttypes() { return m_filter->evttypes(); } +gen_event_formatter::gen_event_formatter() +{ +} + +gen_event_formatter::~gen_event_formatter() +{ +} + +gen_event_formatter_factory::gen_event_formatter_factory() +{ +} + +gen_event_formatter_factory::~gen_event_formatter_factory() +{ +} diff --git a/userspace/libsinsp/gen_filter.h b/userspace/libsinsp/gen_filter.h index 8aa35c64c..3451c6c98 100644 --- a/userspace/libsinsp/gen_filter.h +++ b/userspace/libsinsp/gen_filter.h @@ -18,6 +18,8 @@ along with Falco. If not, see . #include #include +#include +#include #include #include @@ -273,3 +275,45 @@ class gen_event_filter_factory // Return the set of fields supported by this factory virtual std::list get_fields() = 0; }; + +class gen_event_formatter +{ +public: + enum output_format { + OF_NORMAL = 0, + OF_JSON = 1 + }; + + gen_event_formatter(); + virtual ~gen_event_formatter(); + + virtual void set_format(output_format of, const std::string &format) = 0; + + // Format the output string with the configured format + virtual bool tostring(gen_event *evt, std::string &output) = 0; + + // In some cases, it may be useful to format an output string + // with a custom format. + virtual bool tostring_withformat(gen_event *evt, std::string &output, output_format of) = 0; + + // The map should map from field name, without the '%' + // (e.g. "proc.name"), to field value (e.g. "nginx") + virtual bool get_field_values(gen_event *evt, std::map &fields) = 0; + + virtual output_format get_output_format() = 0; +}; + + +class gen_event_formatter_factory +{ +public: + gen_event_formatter_factory(); + virtual ~gen_event_formatter_factory(); + + // This should be called before any calls to + // create_formatter(), and changes the output format of new + // formatters. + virtual void set_output_format(gen_event_formatter::output_format of) = 0; + + virtual std::shared_ptr create_formatter(const std::string &format) = 0; +}; From ce142f4757d4c2c6ae53cbad2e13d052ba6897c6 Mon Sep 17 00:00:00 2001 From: Mark Stemm Date: Mon, 4 Oct 2021 20:36:55 -0700 Subject: [PATCH 019/148] Use error value over lua_error() in lua callbacks Instead of returning errors in lua callbacks via lua_error(), which stops the lua interpreter, return the error explicitly as a string error or nil. This allows for more graceful error handling on the lua side. Signed-off-by: Mark Stemm --- userspace/chisel/lua_parser_api.cpp | 21 +++++++++++++-------- 1 file changed, 13 insertions(+), 8 deletions(-) diff --git a/userspace/chisel/lua_parser_api.cpp b/userspace/chisel/lua_parser_api.cpp index 1c5e00365..5a7e13cb5 100644 --- a/userspace/chisel/lua_parser_api.cpp +++ b/userspace/chisel/lua_parser_api.cpp @@ -143,10 +143,11 @@ int lua_parser_cbacks::nest(lua_State *ls) catch (const std::exception& e) { lua_pushstring(ls, e.what()); - lua_error(ls); + return 1; } - return 0; + lua_pushnil(ls); + return 1; } int lua_parser_cbacks::unnest(lua_State *ls) @@ -167,10 +168,11 @@ int lua_parser_cbacks::unnest(lua_State *ls) catch (const std::exception& e) { lua_pushstring(ls, e.what()); - lua_error(ls); + return 1; } - return 0; + lua_pushnil(ls); + return 1; } int lua_parser_cbacks::bool_op(lua_State *ls) @@ -211,9 +213,11 @@ int lua_parser_cbacks::bool_op(lua_State *ls) catch (const std::exception& e) { lua_pushstring(ls, e.what()); - lua_error(ls); + return 1; } - return 0; + + lua_pushnil(ls); + return 1; } @@ -301,9 +305,10 @@ int lua_parser_cbacks::rel_expr(lua_State *ls) catch (const std::exception& e) { lua_pushstring(ls, e.what()); - lua_error(ls); + return 1; } - return 0; + lua_pushnil(ls); + return 1; } From 55b2be3d9bf1322faf1313a7f7b085e6165f82d8 Mon Sep 17 00:00:00 2001 From: Mark Stemm Date: Thu, 7 Oct 2021 15:19:42 -0700 Subject: [PATCH 020/148] Also skip EF_OLD_VERSION events from simple consumer The event table has a few "old" versions of some events, where the libs needed new params for an event, and did that by defining a new event. PPM_SYSCALL_EXECVE_* are good examples, where the parameters kept changing as we wanted to track additional info during execs. There's no reason to consider these old events with a simple consumer, so exclude events with this flag. Signed-off-by: Mark Stemm --- userspace/libsinsp/sinsp.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/userspace/libsinsp/sinsp.h b/userspace/libsinsp/sinsp.h index f54e5bb66..595c2ec5e 100644 --- a/userspace/libsinsp/sinsp.h +++ b/userspace/libsinsp/sinsp.h @@ -929,7 +929,7 @@ VISIBILITY_PRIVATE static inline ppm_event_flags simple_consumer_skip_flags() { - return (ppm_event_flags) (EF_SKIPPARSERESET | EF_UNUSED | EF_DROP_SIMPLE_CONS); + return (ppm_event_flags) (EF_SKIPPARSERESET | EF_UNUSED | EF_DROP_SIMPLE_CONS | EF_OLD_VERSION); } // Doxygen doesn't understand VISIBILITY_PRIVATE #ifdef _DOXYGEN From 7f84dd1f3bd91861fd00f95d429404af2d760276 Mon Sep 17 00:00:00 2001 From: Mark Stemm Date: Thu, 7 Oct 2021 15:25:05 -0700 Subject: [PATCH 021/148] Set EF_OLD_VERSION for PPME_CONTAINER_{E,X} These events are replaced by PPME_CONTAINER_JSON_{E,X}. Signed-off-by: Mark Stemm --- driver/event_table.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/driver/event_table.c b/driver/event_table.c index 03fee8f28..50567c27c 100644 --- a/driver/event_table.c +++ b/driver/event_table.c @@ -240,8 +240,8 @@ const struct ppm_event_info g_event_info[PPM_EVENT_MAX] = { /* PPME_SYSCALL_FORK_20_X */{"fork", EC_PROCESS, EF_MODIFIES_STATE, 20, {{"res", PT_PID, PF_DEC}, {"exe", PT_CHARBUF, PF_NA}, {"args", PT_BYTEBUF, PF_NA}, {"tid", PT_PID, PF_DEC}, {"pid", PT_PID, PF_DEC}, {"ptid", PT_PID, PF_DEC}, {"cwd", PT_CHARBUF, PF_NA}, {"fdlimit", PT_INT64, PF_DEC}, {"pgft_maj", PT_UINT64, PF_DEC}, {"pgft_min", PT_UINT64, PF_DEC}, {"vm_size", PT_UINT32, PF_DEC}, {"vm_rss", PT_UINT32, PF_DEC}, {"vm_swap", PT_UINT32, PF_DEC}, {"comm", PT_CHARBUF, PF_NA}, {"cgroups", PT_BYTEBUF, PF_NA}, {"flags", PT_FLAGS32, PF_HEX, clone_flags}, {"uid", PT_UINT32, PF_DEC}, {"gid", PT_UINT32, PF_DEC}, {"vtid", PT_PID, PF_DEC}, {"vpid", PT_PID, PF_DEC} } }, /* PPME_SYSCALL_VFORK_20_E */{"vfork", EC_PROCESS, EF_MODIFIES_STATE, 0}, /* PPME_SYSCALL_VFORK_20_X */{"vfork", EC_PROCESS, EF_MODIFIES_STATE, 20, {{"res", PT_PID, PF_DEC}, {"exe", PT_CHARBUF, PF_NA}, {"args", PT_BYTEBUF, PF_NA}, {"tid", PT_PID, PF_DEC}, {"pid", PT_PID, PF_DEC}, {"ptid", PT_PID, PF_DEC}, {"cwd", PT_CHARBUF, PF_NA}, {"fdlimit", PT_INT64, PF_DEC}, {"pgft_maj", PT_UINT64, PF_DEC}, {"pgft_min", PT_UINT64, PF_DEC}, {"vm_size", PT_UINT32, PF_DEC}, {"vm_rss", PT_UINT32, PF_DEC}, {"vm_swap", PT_UINT32, PF_DEC}, {"comm", PT_CHARBUF, PF_NA}, {"cgroups", PT_BYTEBUF, PF_NA}, {"flags", PT_FLAGS32, PF_HEX, clone_flags}, {"uid", PT_UINT32, PF_DEC}, {"gid", PT_UINT32, PF_DEC}, {"vtid", PT_PID, PF_DEC}, {"vpid", PT_PID, PF_DEC} } }, - /* PPME_CONTAINER_E */{"container", EC_INTERNAL, EF_SKIPPARSERESET | EF_MODIFIES_STATE, 4, {{"id", PT_CHARBUF, PF_NA}, {"type", PT_UINT32, PF_DEC}, {"name", PT_CHARBUF, PF_NA}, {"image", PT_CHARBUF, PF_NA} } }, - /* PPME_CONTAINER_X */{"container", EC_INTERNAL, EF_UNUSED, 0}, + /* PPME_CONTAINER_E */{"container", EC_INTERNAL, EF_SKIPPARSERESET | EF_MODIFIES_STATE | EF_OLD_VERSION, 4, {{"id", PT_CHARBUF, PF_NA}, {"type", PT_UINT32, PF_DEC}, {"name", PT_CHARBUF, PF_NA}, {"image", PT_CHARBUF, PF_NA} } }, + /* PPME_CONTAINER_X */{"container", EC_INTERNAL, EF_UNUSED | EF_OLD_VERSION, 0}, /* PPME_SYSCALL_EXECVE_16_E */{"execve", EC_PROCESS, EF_MODIFIES_STATE, 0}, /* PPME_SYSCALL_EXECVE_16_X */{"execve", EC_PROCESS, EF_MODIFIES_STATE, 16, {{"res", PT_ERRNO, PF_DEC}, {"exe", PT_CHARBUF, PF_NA}, {"args", PT_BYTEBUF, PF_NA}, {"tid", PT_PID, PF_DEC}, {"pid", PT_PID, PF_DEC}, {"ptid", PT_PID, PF_DEC}, {"cwd", PT_CHARBUF, PF_NA}, {"fdlimit", PT_UINT64, PF_DEC}, {"pgft_maj", PT_UINT64, PF_DEC}, {"pgft_min", PT_UINT64, PF_DEC}, {"vm_size", PT_UINT32, PF_DEC}, {"vm_rss", PT_UINT32, PF_DEC}, {"vm_swap", PT_UINT32, PF_DEC}, {"comm", PT_CHARBUF, PF_NA}, {"cgroups", PT_BYTEBUF, PF_NA}, {"env", PT_BYTEBUF, PF_NA} } }, /* PPME_SIGNALDELIVER_E */ {"signaldeliver", EC_SIGNAL, EF_DROP_SIMPLE_CONS, 3, {{"spid", PT_PID, PF_DEC}, {"dpid", PT_PID, PF_DEC}, {"sig", PT_SIGTYPE, PF_DEC} } }, From 0a34d5460cf3fb55820922e197c5048e25546055 Mon Sep 17 00:00:00 2001 From: Mark Stemm Date: Thu, 7 Oct 2021 15:49:12 -0700 Subject: [PATCH 022/148] Skip deprecated events in evttypes() When iterating over event numbers (e.g. PPME_EVENT_CONTAINER_{E,X}) to see if they match a param that has the event number as a string (e.g. "container"), skip all events that are old (e.g. EF_UNUSED or EF_OLD_VERSION). Signed-off-by: Mark Stemm --- userspace/libsinsp/filter.cpp | 10 ++++++++++ userspace/libsinsp/filterchecks.cpp | 7 +++++++ 2 files changed, 17 insertions(+) diff --git a/userspace/libsinsp/filter.cpp b/userspace/libsinsp/filter.cpp index 8ae9aa207..66833de67 100644 --- a/userspace/libsinsp/filter.cpp +++ b/userspace/libsinsp/filter.cpp @@ -655,11 +655,21 @@ sinsp_filter_check::sinsp_filter_check() // Do this once if(s_all_event_types.size() == 0) { + sinsp_evttables* einfo = m_inspector->get_event_info_tables(); + const struct ppm_event_info* etable = einfo->m_event_info; + // // Fill in from 2 to PPM_EVENT_MAX-1. 0 and 1 are excluded as // those are PPM_GENERIC_E/PPME_GENERIC_X. for(uint16_t i = 2; i < PPM_EVENT_MAX; i++) { + // Skip "old" event versions that have been replaced + // by newer event versions, or events that are unused. + if(etable[i].flags & (EF_OLD_VERSION | EF_UNUSED)) + { + continue; + } + s_all_event_types.insert(i); } } diff --git a/userspace/libsinsp/filterchecks.cpp b/userspace/libsinsp/filterchecks.cpp index ba053529e..132c6d5a1 100644 --- a/userspace/libsinsp/filterchecks.cpp +++ b/userspace/libsinsp/filterchecks.cpp @@ -4564,6 +4564,13 @@ const std::set &sinsp_filter_check_event::evttypes() // operators/inverse don't work for these values. for(uint32_t i = 2; i < PPM_EVENT_MAX; i++) { + // Skip "old" event versions that have been replaced + // by newer event versions, or events that are unused. + if(etable[i].flags & (EF_OLD_VERSION | EF_UNUSED)) + { + continue; + } + // The values are held as strings, so we need to // convert them back to numbers. bool found = false; From a03ccfda795f2ba711b80f69cb06869f2b63121b Mon Sep 17 00:00:00 2001 From: Mark Stemm Date: Thu, 7 Oct 2021 17:00:23 -0700 Subject: [PATCH 023/148] Update tests to reflect skipping deprecated events Events with EF_OLD_VERSION/EF_UNUSED are skipped by libsinsp, so skip them here too. Signed-off-by: Mark Stemm --- userspace/libsinsp/test/evttype_filter.ut.cpp | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/userspace/libsinsp/test/evttype_filter.ut.cpp b/userspace/libsinsp/test/evttype_filter.ut.cpp index 7b9027490..45fe323ae 100644 --- a/userspace/libsinsp/test/evttype_filter.ut.cpp +++ b/userspace/libsinsp/test/evttype_filter.ut.cpp @@ -19,6 +19,8 @@ limitations under the License. #include #include +extern sinsp_evttables g_infotables; + std::stringstream & operator<<(std::stringstream &out, set s) { out << "[ "; @@ -41,6 +43,13 @@ class evttype_filter_test : public testing::Test { for(uint32_t i = 2; i < PPM_EVENT_MAX; i++) { + // Skip "old" event versions that have been replaced + // by newer event versions, or events that are unused. + if(g_infotables.m_event_info[i].flags & (EF_OLD_VERSION | EF_UNUSED)) + { + continue; + } + all_events.insert(i); if(openat_only.find(i) == openat_only.end()) @@ -102,7 +111,6 @@ class evttype_filter_test : public testing::Test } std::set openat_only{ - PPME_SYSCALL_OPENAT_E, PPME_SYSCALL_OPENAT_X, PPME_SYSCALL_OPENAT_2_E, PPME_SYSCALL_OPENAT_2_X }; @@ -111,7 +119,6 @@ class evttype_filter_test : public testing::Test }; std::set openat_close{ - PPME_SYSCALL_OPENAT_E, PPME_SYSCALL_OPENAT_X, PPME_SYSCALL_OPENAT_2_E, PPME_SYSCALL_OPENAT_2_X, PPME_SYSCALL_CLOSE_E, PPME_SYSCALL_CLOSE_X }; From 326b87e5d7e02b86196863081996f8e0391bf734 Mon Sep 17 00:00:00 2001 From: Grzegorz Nosek Date: Fri, 15 Oct 2021 19:38:41 +0200 Subject: [PATCH 024/148] RHEL 8 backports actually happened in 8.1, not 8.0 This fixes eBPF build for RHEL 8.0 kernels Signed-off-by: Grzegorz Nosek --- driver/bpf/fillers.h | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/driver/bpf/fillers.h b/driver/bpf/fillers.h index 2bf292523..2326bf28f 100644 --- a/driver/bpf/fillers.h +++ b/driver/bpf/fillers.h @@ -1395,7 +1395,7 @@ static __always_inline int bpf_ppm_get_tty(struct task_struct *task) static __always_inline struct pid *bpf_task_pid(struct task_struct *task) { -#if (PPM_RHEL_RELEASE_CODE > 0 && PPM_RHEL_RELEASE_CODE >= PPM_RHEL_RELEASE_VERSION(8, 0)) +#if (PPM_RHEL_RELEASE_CODE > 0 && PPM_RHEL_RELEASE_CODE >= PPM_RHEL_RELEASE_VERSION(8, 1)) return _READ(task->thread_pid); #elif LINUX_VERSION_CODE < KERNEL_VERSION(4, 19, 0) return _READ(task->pids[PIDTYPE_PID].pid); @@ -1434,7 +1434,7 @@ static __always_inline pid_t bpf_pid_nr_ns(struct pid *pid, return nr; } -#if ((PPM_RHEL_RELEASE_CODE > 0 && PPM_RHEL_RELEASE_CODE >= PPM_RHEL_RELEASE_VERSION(8, 0))) || LINUX_VERSION_CODE >= KERNEL_VERSION(4, 19, 0) +#if ((PPM_RHEL_RELEASE_CODE > 0 && PPM_RHEL_RELEASE_CODE >= PPM_RHEL_RELEASE_VERSION(8, 1))) || LINUX_VERSION_CODE >= KERNEL_VERSION(4, 19, 0) static __always_inline struct pid **bpf_task_pid_ptr(struct task_struct *task, enum pid_type type) { @@ -1453,7 +1453,7 @@ static __always_inline pid_t bpf_task_pid_nr_ns(struct task_struct *task, if (!ns) ns = bpf_task_active_pid_ns(task); -#if (PPM_RHEL_RELEASE_CODE > 0 && PPM_RHEL_RELEASE_CODE >= PPM_RHEL_RELEASE_VERSION(8, 0)) +#if (PPM_RHEL_RELEASE_CODE > 0 && PPM_RHEL_RELEASE_CODE >= PPM_RHEL_RELEASE_VERSION(8, 1)) nr = bpf_pid_nr_ns(_READ(*bpf_task_pid_ptr(task, type)), ns); #elif LINUX_VERSION_CODE < KERNEL_VERSION(4, 19, 0) if (type != PIDTYPE_PID) { @@ -1478,7 +1478,7 @@ static __always_inline pid_t bpf_task_pid_vnr(struct task_struct *task) static __always_inline pid_t bpf_task_tgid_vnr(struct task_struct *task) { -#if (PPM_RHEL_RELEASE_CODE > 0 && PPM_RHEL_RELEASE_CODE >= PPM_RHEL_RELEASE_VERSION(8, 0)) +#if (PPM_RHEL_RELEASE_CODE > 0 && PPM_RHEL_RELEASE_CODE >= PPM_RHEL_RELEASE_VERSION(8, 1)) return bpf_task_pid_nr_ns(task, PIDTYPE_TGID, NULL); #elif LINUX_VERSION_CODE < KERNEL_VERSION(4, 19, 0) return bpf_task_pid_nr_ns(task, __PIDTYPE_TGID, NULL); From 769210f4baa1913107ec6e45d088c4baaad03d88 Mon Sep 17 00:00:00 2001 From: Mark Stemm Date: Tue, 5 Oct 2021 13:27:34 -0700 Subject: [PATCH 025/148] Classes to represent a plugin New classes that represent a plugin/filterchecks for plugins. High level summary of classes: sinsp_plugin: base class, an object representing a loaded plugin, handles the dynamic library loading/function resolving/etc. Includes the functions that were dlsym()d from the shared library as well as demographic info (name, desc, contact) read from the plugin via plugin_get_name(), plugin_get_desc(), etc. Has static methods to create a plugin object and register it with sinsp. Note that sinsp_plugin does *not* have any ability to return events via a next() function. That's handled in libscap. sinsp_source_plugin: child of sinsp_plugin, has additional methods to get read progress and display events as a string. Both call the underlying plugin functions. sinsp_extractor_plugin: child of sinsp_plugin, has additional methods to check for a compatible source. sinsp_async_extractor: handles the framework side of plugin_register_async_extractor. In extract, updates the shared struct and waits for a value from the plugin. sinsp_filter_check_plugininfo: filtercheck class that handles the fields "evt.pluginname" and "evt.plugininfo". Each object is associated with a single plugin and the object is created in sinsp_plugin::register_plugin(). sinsp_filter_check_plugin: filtercheck class that handles all other fields exported by a plugin. The set of fields are those exported by the plugin in plugin_get_fields(), and extract calls the plugin's sinsp_plugin::extract_field() method, which in turn calls plugin_extract_fields(). Co-authored-by: Leonardo Grasso Co-authored-by: Loris Degioanni Signed-off-by: Mark Stemm --- userspace/libsinsp/CMakeLists.txt | 1 + userspace/libsinsp/plugin.cpp | 1050 +++++++++++++++++++++++++++++ userspace/libsinsp/plugin.h | 361 ++++++++++ 3 files changed, 1412 insertions(+) create mode 100755 userspace/libsinsp/plugin.cpp create mode 100755 userspace/libsinsp/plugin.h diff --git a/userspace/libsinsp/CMakeLists.txt b/userspace/libsinsp/CMakeLists.txt index 49be45630..cd7fc2820 100644 --- a/userspace/libsinsp/CMakeLists.txt +++ b/userspace/libsinsp/CMakeLists.txt @@ -78,6 +78,7 @@ set(SINSP_SOURCES "${JSONCPP_LIB_SRC}" logger.cpp parsers.cpp + plugin.cpp prefix_search.cpp protodecoder.cpp threadinfo.cpp diff --git a/userspace/libsinsp/plugin.cpp b/userspace/libsinsp/plugin.cpp new file mode 100755 index 000000000..e10658f26 --- /dev/null +++ b/userspace/libsinsp/plugin.cpp @@ -0,0 +1,1050 @@ +/* +Copyright (C) 2021 The Falco Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. + +*/ + +#ifndef _WIN32 +#include +#include +#include +#include +#include +#endif +#include + +#include "sinsp.h" +#include "sinsp_int.h" +#include "filter.h" +#include "filterchecks.h" +#include "plugin.h" +#include "plugin_evt_processor.h" + +#include + +using namespace std; + +extern sinsp_filter_check_list g_filterlist; + +/////////////////////////////////////////////////////////////////////////////// +// source_plugin filter check implementation +// This class implements a dynamic filter check that acts as a bridge to the +// plugin simplified field extraction implementations +/////////////////////////////////////////////////////////////////////////////// + +const filtercheck_field_info sinsp_filter_check_plugininfo_fields[] = +{ + {PT_CHARBUF, EPF_NONE, PF_NA, "evt.pluginname", "if the event comes from a plugin, the name of the plugin that generated it."}, + {PT_CHARBUF, EPF_NONE, PF_NA, "evt.plugininfo", "if the event comes from a plugin, a summary of the event as formatted by the plugin."}, +}; + +class sinsp_filter_check_plugininfo : public sinsp_filter_check +{ +public: + enum check_type + { + TYPE_PLUGINNAME = 0, + TYPE_PLUGININFO = 1, + }; + + sinsp_filter_check_plugininfo() + { + m_info.m_name = "plugininfo"; + m_info.m_fields = sinsp_filter_check_plugininfo_fields; + m_info.m_nfields = sizeof(sinsp_filter_check_plugininfo_fields) / sizeof(sinsp_filter_check_plugininfo_fields[0]); + m_info.m_flags = filter_check_info::FL_NONE; + } + + sinsp_filter_check_plugininfo(std::shared_ptr plugin) + : m_plugin(plugin) + { + m_info.m_name = plugin->name() + string(" (plugininfo)"); + m_info.m_fields = sinsp_filter_check_plugininfo_fields; + m_info.m_nfields = sizeof(sinsp_filter_check_plugininfo_fields) / sizeof(sinsp_filter_check_plugininfo_fields[0]); + m_info.m_flags = filter_check_info::FL_NONE; + } + + sinsp_filter_check_plugininfo(const sinsp_filter_check_plugininfo &p) + { + m_plugin = p.m_plugin; + m_info = p.m_info; + } + + virtual ~sinsp_filter_check_plugininfo() + { + } + + sinsp_filter_check* allocate_new() + { + return new sinsp_filter_check_plugininfo(*this); + } + + uint8_t* extract(sinsp_evt *evt, OUT uint32_t* len, bool sanitize_strings) + { + // + // Only extract if the event is a plugin event and if + // this plugin is a source plugin. + // + if(!(evt->get_type() == PPME_PLUGINEVENT_E && + m_plugin->type() == TYPE_SOURCE_PLUGIN)) + { + return NULL; + } + + // + // Only extract if the event plugin id matches this plugin's id. + // + sinsp_source_plugin *splugin = static_cast(m_plugin.get()); + + sinsp_evt_param *parinfo; + parinfo = evt->get_param(0); + ASSERT(parinfo->m_len == sizeof(int32_t)); + uint32_t pgid = *(int32_t *)parinfo->m_val; + if(pgid != splugin->id()) + { + return NULL; + } + + switch(m_field_id) + { + case TYPE_PLUGINNAME: + m_strstorage = splugin->name(); + *len = m_strstorage.size(); + return (uint8_t*) m_strstorage.c_str(); + break; + case TYPE_PLUGININFO: + parinfo = evt->get_param(1); + m_strstorage = splugin->event_to_string((const uint8_t *) parinfo->m_val, parinfo->m_len); + *len = m_strstorage.size(); + return (uint8_t*) m_strstorage.c_str(); + default: + return NULL; + } + + return NULL; + } + + std::string m_strstorage; + + std::shared_ptr m_plugin; +}; + +class sinsp_filter_check_plugin : public sinsp_filter_check +{ +public: + sinsp_filter_check_plugin() + { + m_info.m_name = "plugin"; + m_info.m_fields = NULL; + m_info.m_nfields = 0; + m_info.m_flags = filter_check_info::FL_NONE; + m_cnt = 0; + } + + sinsp_filter_check_plugin(std::shared_ptr plugin) + : m_plugin(plugin) + { + m_info.m_name = plugin->name() + string(" (plugin)"); + m_info.m_fields = plugin->fields(); + m_info.m_nfields = plugin->nfields(); + m_info.m_flags = filter_check_info::FL_NONE; + m_cnt = 0; + } + + sinsp_filter_check_plugin(const sinsp_filter_check_plugin &p) + { + m_plugin = p.m_plugin; + m_info = p.m_info; + } + + virtual ~sinsp_filter_check_plugin() + { + } + + int32_t parse_field_name(const char* str, bool alloc_state, bool needed_for_filtering) + { + int32_t res = sinsp_filter_check::parse_field_name(str, alloc_state, needed_for_filtering); + + if(res != -1) + { + // Read from str to the end-of-string, or first space + string val(str); + size_t val_end = val.find_first_of(' ', 0); + if(val_end != string::npos) + { + val = val.substr(0, val_end); + } + + size_t pos1 = val.find_first_of('[', 0); + if(pos1 != string::npos) + { + size_t argstart = pos1 + 1; + if(argstart < val.size()) + { + m_argstr = val.substr(argstart); + size_t pos2 = m_argstr.find_first_of(']', 0); + m_argstr = m_argstr.substr(0, pos2); + m_arg = (char*)m_argstr.c_str(); + return pos1 + pos2 + 2; + } + } + } + + return res; + } + + sinsp_filter_check* allocate_new() + { + return new sinsp_filter_check_plugin(*this); + } + + uint8_t* extract(sinsp_evt *evt, OUT uint32_t* len, bool sanitize_strings) + { + // + // Reject any event that is not generated by a plugin + // + if(evt->get_type() != PPME_PLUGINEVENT_E) + { + return NULL; + } + + // + // If this is a source plugin, reject events that have + // not been generated by a plugin with this id specifically. + // + // XXX/mstemm this should probably check the version as well. + // + sinsp_evt_param *parinfo; + if(m_plugin->type() == TYPE_SOURCE_PLUGIN) + { + sinsp_source_plugin *splugin = static_cast(m_plugin.get()); + parinfo = evt->get_param(0); + ASSERT(parinfo->m_len == sizeof(int32_t)); + uint32_t pgid = *(int32_t *)parinfo->m_val; + if(pgid != splugin->id()) + { + return NULL; + } + } + + // + // If this is an extractor plugin, only attempt to + // extract if the source is compatible with the event + // source. + // + if(m_plugin->type() == TYPE_EXTRACTOR_PLUGIN) + { + sinsp_extractor_plugin *eplugin = static_cast(m_plugin.get()); + parinfo = evt->get_param(0); + ASSERT(parinfo->m_len == sizeof(int32_t)); + uint32_t pgid = *(int32_t *)parinfo->m_val; + + std::shared_ptr plugin = m_inspector->get_plugin_by_id(pgid); + + if(!plugin) + { + return NULL; + } + + sinsp_source_plugin *splugin = static_cast(plugin.get()); + + if(!eplugin->source_compatible(splugin->event_source())) + { + return NULL; + } + } + + // + // Get the event payload + // + parinfo = evt->get_param(1); + *len = 0; + + ppm_param_type type = m_info.m_fields[m_field_id].m_type; + + ss_plugin_event pevt; + pevt.evtnum = evt->get_num(); + pevt.data = (uint8_t *) parinfo->m_val; + pevt.datalen = parinfo->m_len; + pevt.ts = evt->get_ts(); + + sinsp_plugin::ext_field field; + field.field = m_info.m_fields[m_field_id].m_name; + if(m_arg != NULL) + { + field.arg = m_arg; + } + field.ftype = type; + + if (!m_plugin->extract_field(pevt, field) || + ! field.field_present) + { + return NULL; + } + + switch(type) + { + case PT_CHARBUF: + { + m_strstorage = field.res_str; + *len = m_strstorage.size(); + return (uint8_t*) m_strstorage.c_str(); + } + case PT_UINT64: + { + m_u64_res = field.res_u64; + return (uint8_t *)&m_u64_res; + } + default: + ASSERT(false); + throw sinsp_exception("plugin extract error: unsupported field type " + to_string(type)); + break; + } + + return NULL; + } + + // XXX/mstemm m_cnt unused so far. + uint64_t m_cnt; + string m_argstr; + char* m_arg = NULL; + + std::string m_strstorage; + uint64_t m_u64_res; + + std::shared_ptr m_plugin; +}; + +/////////////////////////////////////////////////////////////////////////////// +// sinsp_plugin implementation +/////////////////////////////////////////////////////////////////////////////// +sinsp_plugin::version::version() + : m_valid(false) +{ +} + +sinsp_plugin::version::version(const std::string &version_str) + : m_valid(false) +{ + m_valid = (sscanf(version_str.c_str(), "%" PRIu32 ".%" PRIu32 ".%" PRIu32, + &m_version_major, &m_version_minor, &m_version_patch) == 3); +} + +sinsp_plugin::version::~version() +{ +} + +std::string sinsp_plugin::version::as_string() const +{ + return std::to_string(m_version_major) + "." + + std::to_string(m_version_minor) + "." + + std::to_string(m_version_patch); +} + +std::shared_ptr sinsp_plugin::register_plugin(sinsp* inspector, string filepath, char* config, bool avoid_async) +{ + string errstr; + std::shared_ptr plugin = create_plugin(filepath, config, avoid_async, errstr); + + if (!plugin) + { + throw sinsp_exception("cannot load plugin " + filepath + ": " + errstr.c_str()); + } + + try + { + inspector->add_plugin(plugin); + } + catch(sinsp_exception const& e) + { + throw sinsp_exception("cannot add plugin " + filepath + " to inspector: " + e.what()); + } + + // + // Create and register the filter checks associated to this plugin + // + auto info_filtercheck = new sinsp_filter_check_plugininfo(plugin); + g_filterlist.add_filter_check(info_filtercheck); + + auto filtercheck = new sinsp_filter_check_plugin(plugin); + g_filterlist.add_filter_check(filtercheck); + + return plugin; +} + +std::shared_ptr sinsp_plugin::create_plugin(string &filepath, char* config, bool avoid_async, std::string &errstr) +{ + std::shared_ptr ret; + +#ifdef _WIN32 + HINSTANCE handle = LoadLibrary(filepath.c_str()); +#else + void* handle = dlopen(filepath.c_str(), RTLD_LAZY); +#endif + if(handle == NULL) + { + errstr = "error loading plugin " + filepath + ": " + strerror(errno); + return ret; + } + + // Before doing anything else, check the required api + // version. If it doesn't match, return an error. + + // The pointer indirection and reference is because c++ doesn't + // strictly allow casting void * to a function pointer. (See + // http://www.open-std.org/jtc1/sc22/wg21/docs/cwg_defects.html#195). + char * (*get_required_api_version)(); + *(void **) (&get_required_api_version) = getsym(handle, "plugin_get_required_api_version", errstr); + if(get_required_api_version == NULL) + { + errstr = string("Could not resolve plugin_get_required_api_version function"); + return ret; + } + + std::string version_str = str_from_alloc_charbuf(get_required_api_version()); + version v(version_str); + if(!v.m_valid) + { + errstr = string("Could not parse version string from ") + version_str; + return ret; + } + + if(v.m_version_major != PLUGIN_API_VERSION_MAJOR) + { + errstr = string("Unsupported plugin required api version ") + version_str; + return ret; + } + + uint32_t (*get_type)(); + *(void **) (&get_type) = getsym(handle, "plugin_get_type", errstr); + if(get_type == NULL) + { + errstr = string("Could not resolve plugin_get_type function"); + return ret; + } + + uint32_t plugin_type = get_type(); + + sinsp_source_plugin *splugin; + sinsp_extractor_plugin *eplugin; + + switch(plugin_type) + { + case TYPE_SOURCE_PLUGIN: + splugin = new sinsp_source_plugin(); + if(!splugin->resolve_dylib_symbols(handle, errstr)) + { + delete splugin; + return ret; + } + ret.reset(splugin); + break; + case TYPE_EXTRACTOR_PLUGIN: + eplugin = new sinsp_extractor_plugin(); + if(!eplugin->resolve_dylib_symbols(handle, errstr)) + { + delete eplugin; + return ret; + } + ret.reset(eplugin); + break; + } + + errstr = ""; + + // Initialize the plugin + if (!ret->init(config, avoid_async)) + { + ret = NULL; + } + + return ret; +} + +std::list sinsp_plugin::plugin_infos(sinsp* inspector) +{ + std::list ret; + + for(auto p : inspector->get_plugins()) + { + sinsp_plugin::info info; + info.name = p->name(); + info.description = p->description(); + info.contact = p->contact(); + info.plugin_version = p->plugin_version(); + info.required_api_version = p->required_api_version(); + + if(p->type() == TYPE_SOURCE_PLUGIN) + { + sinsp_source_plugin *sp = static_cast(p.get()); + info.id = sp->id(); + } + ret.push_back(info); + } + + return ret; +} + +sinsp_plugin::sinsp_plugin() + : m_nfields(0) +{ +} + +sinsp_plugin::~sinsp_plugin() +{ +} + +bool sinsp_plugin::init(char *config, bool avoid_async) +{ + if (!m_plugin_info.init) + { + return false; + } + + int32_t rc; + + ss_plugin_t *state = m_plugin_info.init(config, &rc); + if(rc != SCAP_SUCCESS) + { + // Not calling get_last_error here because there was + // no valid ss_plugin_t struct returned from init. + return false; + } + + set_plugin_state(state); + + if(m_plugin_info.register_async_extractor && !avoid_async) + { + m_async_extractor.reset(new sinsp_async_extractor()); + + if(m_plugin_info.register_async_extractor(plugin_state(), m_async_extractor->extractor_info()) != SCAP_SUCCESS) + { + throw sinsp_exception(string("error in plugin ") + m_name + ": " + get_last_error()); + } + } + + return true; +} + +void sinsp_plugin::destroy() +{ + if(plugin_state() && m_plugin_info.destroy) + { + m_plugin_info.destroy(plugin_state()); + set_plugin_state(NULL); + } +} + +std::string sinsp_plugin::get_last_error() +{ + std::string ret; + + if(plugin_state() && m_plugin_info.get_last_error) + { + ret = str_from_alloc_charbuf(m_plugin_info.get_last_error(plugin_state())); + } + else + { + ret = "Plugin handle or get_last_error function not defined"; + } + + return ret; +} + +const std::string &sinsp_plugin::name() +{ + return m_name; +} + +const std::string &sinsp_plugin::description() +{ + return m_description; +} + +const std::string &sinsp_plugin::contact() +{ + return m_contact; +} + +const sinsp_plugin::version &sinsp_plugin::plugin_version() +{ + return m_plugin_version; +} + +const sinsp_plugin::version &sinsp_plugin::required_api_version() +{ + return m_required_api_version; +} + +const filtercheck_field_info *sinsp_plugin::fields() +{ + return m_fields.get(); +} + +uint32_t sinsp_plugin::nfields() +{ + return m_nfields; +} + +bool sinsp_plugin::extract_field(ss_plugin_event &evt, sinsp_plugin::ext_field &field) +{ + if(!m_plugin_info.extract_fields || !plugin_state()) + { + return false; + } + + uint32_t num_fields = 1; + ss_plugin_extract_field efield; + efield.field = field.field.c_str(); + efield.arg = field.arg.c_str(); + efield.ftype = field.ftype; + + int32_t rc; + + if(m_async_extractor) + { + rc = m_async_extractor->extract_field(evt, efield); + } + else + { + rc = m_plugin_info.extract_fields(plugin_state(), &evt, num_fields, &efield); + } + + if (rc != SCAP_SUCCESS) + { + return false; + } + + field.field_present = efield.field_present; + switch(field.ftype) + { + case PT_CHARBUF: + field.res_str = str_from_alloc_charbuf(efield.res_str); + break; + case PT_UINT64: + field.res_u64 = efield.res_u64; + break; + default: + ASSERT(false); + throw sinsp_exception("plugin extract error: unsupported field type " + to_string(field.ftype)); + break; + } + + return true; +} + +void* sinsp_plugin::getsym(void* handle, const char* name, std::string &errstr) +{ + void *ret; + +#ifdef _WIN32 + ret = GetProcAddress((HINSTANCE)handle, name); +#else + ret = dlsym(handle, name); +#endif + + if(ret == NULL) + { + errstr = string("Dynamic library symbol ") + name + " not present"; + } else { + errstr = ""; + } + + return ret; +} + +// Used below--set a std::string from the provided allocated charbuf and free() the charbuf. +std::string sinsp_plugin::str_from_alloc_charbuf(char *charbuf) +{ + std::string str; + + if(charbuf != NULL) + { + str = charbuf; + free(charbuf); + } + + return str; +} + +bool sinsp_plugin::resolve_dylib_symbols(void *handle, std::string &errstr) +{ + // Some functions are required and return false if not found. + if((*(void **) (&(m_plugin_info.get_required_api_version)) = getsym(handle, "plugin_get_required_api_version", errstr)) == NULL || + (*(void **) (&(m_plugin_info.get_last_error)) = getsym(handle, "plugin_get_last_error", errstr)) == NULL || + (*(void **) (&(m_plugin_info.get_name)) = getsym(handle, "plugin_get_name", errstr)) == NULL || + (*(void **) (&(m_plugin_info.get_description)) = getsym(handle, "plugin_get_description", errstr)) == NULL || + (*(void **) (&(m_plugin_info.get_contact)) = getsym(handle, "plugin_get_contact", errstr)) == NULL || + (*(void **) (&(m_plugin_info.get_version)) = getsym(handle, "plugin_get_version", errstr)) == NULL) + { + return false; + } + + // Others are not and the values will be checked when needed. + (*(void **) (&m_plugin_info.init)) = getsym(handle, "plugin_init", errstr); + (*(void **) (&m_plugin_info.destroy)) = getsym(handle, "plugin_destroy", errstr); + (*(void **) (&m_plugin_info.get_fields)) = getsym(handle, "plugin_get_fields", errstr); + (*(void **) (&m_plugin_info.extract_fields)) = getsym(handle, "plugin_extract_fields", errstr); + (*(void **) (&m_plugin_info.register_async_extractor)) = getsym(handle, "plugin_register_async_extractor", errstr); + + m_name = str_from_alloc_charbuf(m_plugin_info.get_name()); + m_description = str_from_alloc_charbuf(m_plugin_info.get_description()); + m_contact = str_from_alloc_charbuf(m_plugin_info.get_contact()); + std::string version_str = str_from_alloc_charbuf(m_plugin_info.get_version()); + m_plugin_version = sinsp_plugin::version(version_str); + if(!m_plugin_version.m_valid) + { + errstr = string("Could not parse version string from ") + version_str; + return false; + } + + // The required api version was already checked in + // create_plugin to be valid and compatible. This just saves it for info/debugging. + version_str = str_from_alloc_charbuf(m_plugin_info.get_required_api_version()); + m_required_api_version = sinsp_plugin::version(version_str); + + // + // If filter fields are exported by the plugin, get the json from get_fields(), + // parse it, create our list of fields, and create a filtercheck from the fields. + // + if(m_plugin_info.get_fields) + { + char* sfields = m_plugin_info.get_fields(); + if(sfields == NULL) + { + throw sinsp_exception(string("error in plugin ") + m_name + ": get_fields returned a null string"); + } + string json(sfields); + SINSP_DEBUG("Parsing Fields JSON=%s", json.c_str()); + Json::Value root; + if(Json::Reader().parse(json, root) == false || root.type() != Json::arrayValue) + { + throw sinsp_exception(string("error in plugin ") + m_name + ": get_fields returned an invalid JSON"); + } + + filtercheck_field_info *fields = new filtercheck_field_info[root.size()]; + if(fields == NULL) + { + throw sinsp_exception(string("error in plugin ") + m_name + ": could not allocate memory"); + } + + // Take ownership of the pointer right away so it can't be leaked. + m_fields.reset(fields); + m_nfields = root.size(); + + for(Json::Value::ArrayIndex j = 0; j < root.size(); j++) + { + filtercheck_field_info &tf = m_fields.get()[j]; + tf.m_flags = EPF_NONE; + + const Json::Value &jvtype = root[j]["type"]; + string ftype = jvtype.asString(); + if(ftype == "") + { + throw sinsp_exception(string("error in plugin ") + m_name + ": field JSON entry has no type"); + } + const Json::Value &jvname = root[j]["name"]; + string fname = jvname.asString(); + if(fname == "") + { + throw sinsp_exception(string("error in plugin ") + m_name + ": field JSON entry has no name"); + } + const Json::Value &jvdesc = root[j]["desc"]; + string fdesc = jvdesc.asString(); + if(fdesc == "") + { + throw sinsp_exception(string("error in plugin ") + m_name + ": field JSON entry has no desc"); + } + + strncpy(tf.m_name, fname.c_str(), sizeof(tf.m_name)); + strncpy(tf.m_description, fdesc.c_str(), sizeof(tf.m_description)); + tf.m_print_format = PF_DEC; + if(ftype == "string") + { + tf.m_type = PT_CHARBUF; + } + else if(ftype == "uint64") + { + tf.m_type = PT_UINT64; + } + // XXX/mstemm are these actually supported? + else if(ftype == "int64") + { + tf.m_type = PT_INT64; + } + else if(ftype == "float") + { + tf.m_type = PT_DOUBLE; + } + else + { + throw sinsp_exception(string("error in plugin ") + m_name + ": invalid field type " + ftype); + } + const Json::Value &jvargRequired = root[j].get("argRequired", Json::Value::null); + if (!jvargRequired.isNull()) + { + if (!jvargRequired.isBool()) + { + throw sinsp_exception(string("error in plugin ") + m_name + ": field " + fname + " argRequired property is not boolean "); + } + + if (jvargRequired.asBool() == true) + { + tf.m_flags = filtercheck_field_flags::EPF_REQUIRES_ARGUMENT; + } + } + } + + } + + return true; +} + +void sinsp_plugin::disable_async_extract() +{ + m_async_extractor.reset(); +} + +sinsp_source_plugin::sinsp_source_plugin() +{ + memset(&m_source_plugin_info, 0, sizeof(m_source_plugin_info)); +} + +sinsp_source_plugin::~sinsp_source_plugin() +{ + close(); + destroy(); +} + +uint32_t sinsp_source_plugin::id() +{ + return m_id; +} + +const std::string &sinsp_source_plugin::event_source() +{ + return m_event_source; +} + +source_plugin_info *sinsp_source_plugin::plugin_info() +{ + return &m_source_plugin_info; +} + +bool sinsp_source_plugin::open(char *params, int32_t &rc) +{ + int32_t orc; + + if(!plugin_state()) + { + return false; + } + + m_source_plugin_info.handle = m_source_plugin_info.open(plugin_state(), params, &orc); + + rc = orc; + + return (m_source_plugin_info.handle != NULL); +} + +void sinsp_source_plugin::close() +{ + if(!plugin_state() || !m_source_plugin_info.handle) + { + return; + } + + m_source_plugin_info.close(plugin_state(), m_source_plugin_info.handle); +} + +std::string sinsp_source_plugin::get_progress(uint32_t &progress_pct) +{ + std::string ret; + progress_pct = 0; + + if(!m_source_plugin_info.get_progress || !m_source_plugin_info.handle) + { + return ret; + } + + uint32_t ppct; + ret = str_from_alloc_charbuf(m_source_plugin_info.get_progress(plugin_state(), m_source_plugin_info.handle, &ppct)); + + progress_pct = ppct; + + return ret; +} + +std::string sinsp_source_plugin::event_to_string(const uint8_t *data, uint32_t datalen) +{ + std::string ret = ""; + + if (!m_source_plugin_info.event_to_string) + { + return ret; + } + + ret = str_from_alloc_charbuf(m_source_plugin_info.event_to_string(plugin_state(), data, datalen)); + + return ret; +} + +void sinsp_source_plugin::set_plugin_state(ss_plugin_t *state) +{ + m_source_plugin_info.state = state; +} + +ss_plugin_t *sinsp_source_plugin::plugin_state() +{ + return m_source_plugin_info.state; +} + +bool sinsp_source_plugin::resolve_dylib_symbols(void *handle, std::string &errstr) +{ + if (!sinsp_plugin::resolve_dylib_symbols(handle, errstr)) + { + return false; + } + + // We resolve every symbol, even those that are not actually + // used by this derived class, just to ensure that + // m_source_plugin_info is complete. (The struct can be passed + // down to libscap when reading/writing capture files). + // + // Some functions are required and return false if not found. + if((*(void **) (&(m_source_plugin_info.get_required_api_version)) = getsym(handle, "plugin_get_required_api_version", errstr)) == NULL || + (*(void **) (&(m_source_plugin_info.init)) = getsym(handle, "plugin_init", errstr)) == NULL || + (*(void **) (&(m_source_plugin_info.destroy)) = getsym(handle, "plugin_destroy", errstr)) == NULL || + (*(void **) (&(m_source_plugin_info.get_last_error)) = getsym(handle, "plugin_get_last_error", errstr)) == NULL || + (*(void **) (&(m_source_plugin_info.get_type)) = getsym(handle, "plugin_get_type", errstr)) == NULL || + (*(void **) (&(m_source_plugin_info.get_id)) = getsym(handle, "plugin_get_id", errstr)) == NULL || + (*(void **) (&(m_source_plugin_info.get_name)) = getsym(handle, "plugin_get_name", errstr)) == NULL || + (*(void **) (&(m_source_plugin_info.get_description)) = getsym(handle, "plugin_get_description", errstr)) == NULL || + (*(void **) (&(m_source_plugin_info.get_contact)) = getsym(handle, "plugin_get_contact", errstr)) == NULL || + (*(void **) (&(m_source_plugin_info.get_version)) = getsym(handle, "plugin_get_version", errstr)) == NULL || + (*(void **) (&(m_source_plugin_info.get_event_source)) = getsym(handle, "plugin_get_event_source", errstr)) == NULL || + (*(void **) (&(m_source_plugin_info.open)) = getsym(handle, "plugin_open", errstr)) == NULL || + (*(void **) (&(m_source_plugin_info.close)) = getsym(handle, "plugin_close", errstr)) == NULL || + (*(void **) (&(m_source_plugin_info.next)) = getsym(handle, "plugin_next", errstr)) == NULL || + (*(void **) (&(m_source_plugin_info.event_to_string)) = getsym(handle, "plugin_event_to_string", errstr)) == NULL) + { + return false; + } + + // Others are not. + (*(void **) (&m_source_plugin_info.get_fields)) = getsym(handle, "plugin_get_fields", errstr); + (*(void **) (&m_source_plugin_info.get_progress)) = getsym(handle, "plugin_get_progress", errstr); + (*(void **) (&m_source_plugin_info.event_to_string)) = getsym(handle, "plugin_event_to_string", errstr); + (*(void **) (&m_source_plugin_info.extract_fields)) = getsym(handle, "plugin_extract_fields", errstr); + (*(void **) (&m_source_plugin_info.next_batch)) = getsym(handle, "plugin_next_batch", errstr); + (*(void **) (&m_source_plugin_info.register_async_extractor)) = getsym(handle, "plugin_register_async_extractor", errstr); + + m_id = m_source_plugin_info.get_id(); + m_event_source = str_from_alloc_charbuf(m_source_plugin_info.get_event_source()); + + return true; +} + +sinsp_extractor_plugin::sinsp_extractor_plugin() +{ + memset(&m_extractor_plugin_info, 0, sizeof(m_extractor_plugin_info)); +} + +sinsp_extractor_plugin::~sinsp_extractor_plugin() +{ + destroy(); +} + +const std::set &sinsp_extractor_plugin::extract_event_sources() +{ + return m_extract_event_sources; +} + +bool sinsp_extractor_plugin::source_compatible(const std::string &source) +{ + return(m_extract_event_sources.size() == 0 || + m_extract_event_sources.find(source) != m_extract_event_sources.end()); +} + +void sinsp_extractor_plugin::set_plugin_state(ss_plugin_t *state) +{ + m_extractor_plugin_info.state = state; +} + +ss_plugin_t *sinsp_extractor_plugin::plugin_state() +{ + return m_extractor_plugin_info.state; +} + +bool sinsp_extractor_plugin::resolve_dylib_symbols(void *handle, std::string &errstr) +{ + if (!sinsp_plugin::resolve_dylib_symbols(handle, errstr)) + { + return false; + } + + // We resolve every symbol, even those that are not actually + // used by this derived class, just to ensure that + // m_extractor_plugin_info is complete. (The struct can be passed + // down to libscap when reading/writing capture files). + // + // Some functions are required and return false if not found. + if((*(void **) (&(m_extractor_plugin_info.get_required_api_version)) = getsym(handle, "plugin_get_required_api_version", errstr)) == NULL || + (*(void **) (&(m_extractor_plugin_info.init)) = getsym(handle, "plugin_init", errstr)) == NULL || + (*(void **) (&(m_extractor_plugin_info.destroy)) = getsym(handle, "plugin_destroy", errstr)) == NULL || + (*(void **) (&(m_extractor_plugin_info.get_last_error)) = getsym(handle, "plugin_get_last_error", errstr)) == NULL || + (*(void **) (&(m_extractor_plugin_info.get_type)) = getsym(handle, "plugin_get_type", errstr)) == NULL || + (*(void **) (&(m_extractor_plugin_info.get_name)) = getsym(handle, "plugin_get_name", errstr)) == NULL || + (*(void **) (&(m_extractor_plugin_info.get_description)) = getsym(handle, "plugin_get_description", errstr)) == NULL || + (*(void **) (&(m_extractor_plugin_info.get_contact)) = getsym(handle, "plugin_get_contact", errstr)) == NULL || + (*(void **) (&(m_extractor_plugin_info.get_version)) = getsym(handle, "plugin_get_version", errstr)) == NULL || + (*(void **) (&(m_extractor_plugin_info.get_fields)) = getsym(handle, "plugin_get_fields", errstr)) == NULL || + (*(void **) (&(m_extractor_plugin_info.extract_fields)) = getsym(handle, "plugin_extract_fields", errstr)) == NULL) + { + return false; + } + + // Others are not. + (*(void **) (&m_extractor_plugin_info.get_extract_event_sources)) = getsym(handle, "plugin_get_extract_event_sources", errstr); + + if (m_extractor_plugin_info.get_extract_event_sources != NULL) + { + std::string esources = str_from_alloc_charbuf(m_extractor_plugin_info.get_extract_event_sources()); + + if (esources.length() == 0) + { + throw sinsp_exception(string("error in plugin ") + name() + ": get_extract_event_sources returned an empty string"); + } + + Json::Value root; + if(Json::Reader().parse(esources, root) == false || root.type() != Json::arrayValue) + { + throw sinsp_exception(string("error in plugin ") + name() + ": get_extract_event_sources did not return a json array"); + } + + for(Json::Value::ArrayIndex j = 0; j < root.size(); j++) + { + if(! root[j].isConvertibleTo(Json::stringValue)) + { + throw sinsp_exception(string("error in plugin ") + name() + ": get_extract_event_sources did not return a json array"); + } + + m_extract_event_sources.insert(root[j].asString()); + } + } + + return true; +} + + diff --git a/userspace/libsinsp/plugin.h b/userspace/libsinsp/plugin.h new file mode 100755 index 000000000..57d3fe3d0 --- /dev/null +++ b/userspace/libsinsp/plugin.h @@ -0,0 +1,361 @@ +/* +Copyright (C) 2021 The Falco Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. + +*/ + +#pragma once + +#include +#include +#include +#include +#include +#include + +#include + +class sinsp_filter_check_plugin; + +class sinsp_async_extractor +{ +public: + sinsp_async_extractor() + { + m_lock = state::INIT; + + m_async_extractor_info.cb_wait = [](void *wait_ctx) + { + return static_cast(wait_ctx)->wait(); + }; + + m_async_extractor_info.wait_ctx = this; + } + + ~sinsp_async_extractor() + { + shutdown(); + } + + struct async_extractor_info *extractor_info() + { + return &m_async_extractor_info; + } + + int32_t extract_field(ss_plugin_event &evt, ss_plugin_extract_field &field) + { + return extract(evt, field); + } + +private: + + enum state + { + INIT = 0, + INPUT_READY = 1, + PROCESSING = 2, + DONE = 3, + SHUTDOWN_REQ = 4, + SHUTDOWN_DONE = 5, + }; + + // On success the caller is responsible for free() in res_str + // when ftype == PT_CHARBUF. + inline int32_t extract(ss_plugin_event &evt, ss_plugin_extract_field &field) + { + m_async_extractor_info.evt = &evt; + m_async_extractor_info.field = &field; + + int old_val = state::DONE; + + while(!m_lock.compare_exchange_strong(old_val, state::INPUT_READY)) + { + old_val = state::DONE; + } + + // + // Once INPUT_READY state has been aquired, wait for worker completition + // + while(m_lock != state::DONE); + + // rc now contains the error code for the extraction. + return m_async_extractor_info.rc; + } + + inline bool wait() + { + m_lock = state::DONE; + uint64_t ncycles = 0; + bool sleeping = false; + + // + // Worker has done and now waits for a new input or a shutdown request. + // Note: we busy loop for the first 1ms to guarantee maximum performance. + // After 1ms we start sleeping to conserve CPU. + // + int old_val = state::INPUT_READY; + + auto start_time = chrono::high_resolution_clock::now(); + + while(!m_lock.compare_exchange_strong(old_val, state::PROCESSING)) + { + // shutdown + if(old_val == state::SHUTDOWN_REQ) + { + m_lock = state::SHUTDOWN_DONE; + return false; + } + old_val = state::INPUT_READY; + + if(sleeping) + { + this_thread::sleep_for(chrono::milliseconds(10)); + } + else + { + ncycles++; + if(ncycles >= 100000) + { + auto cur_time = chrono::high_resolution_clock::now(); + auto delta_time = chrono::duration_cast(cur_time - start_time).count(); + if(delta_time > 1000) + { + sleeping = true; + } + else + { + ncycles = 0; + } + } + } + } + return true; + } + + inline void shutdown() + { + // + // Set SHUTDOWN_REQ iff the worker + // + int old_val = state::DONE; + while(m_lock.compare_exchange_strong(old_val, state::SHUTDOWN_REQ)) + { + old_val = state::DONE; + } + + // await shutdown + // XXX/mstemm add a timeout to this + while (m_lock != state::SHUTDOWN_DONE) + ; + } + +private: + // The shared struct to communicate with the plugin side + struct async_extractor_info m_async_extractor_info; + + atomic m_lock; +}; + +// Base class for source/extractor plugins. Can not be created directly. +class sinsp_plugin +{ +public: + class version { + public: + version(); + version(const std::string &version_str); + virtual ~version(); + + std::string as_string() const; + + bool m_valid; + uint32_t m_version_major; + uint32_t m_version_minor; + uint32_t m_version_patch; + }; + + // Contains important info about a plugin, suitable for + // printing or other checks like compatibility. + struct info { + ss_plugin_type type; + std::string name; + std::string description; + std::string contact; + version plugin_version; + version required_api_version; + + // Only filled in for source plugins + uint32_t id; + }; + + // Similar to struct ss_plugin_extract_field, but with c++ + // types to avoid having to track memory allocations. + struct ext_field { + std::string field; + std::string arg; + uint32_t ftype; + + bool field_present; + std::string res_str; + uint64_t res_u64; + }; + + // Create and register a plugin from a shared library pointed + // to by filepath, and add it to the inspector. + // The created sinsp_plugin is returned. + static std::shared_ptr register_plugin(sinsp* inspector, std::string filepath, char *config, bool avoid_async); + + // Create a plugin from the dynamic library at the provided + // path. On error, the shared_ptr will == NULL and errstr is + // set with an error. + static std::shared_ptr create_plugin(std::string &filepath, char *config, bool avoid_async, std::string &errstr); + + // Return a string with names/descriptions/etc of all plugins used by this inspector + static std::list plugin_infos(sinsp *inspector); + + sinsp_plugin(); + virtual ~sinsp_plugin(); + + // Given a dynamic library handle, fill in common properties + // (name/desc/etc) and required functions + // (init/destroy/extract/etc). + // Returns true on success, false + sets errstr on error. + virtual bool resolve_dylib_symbols(void *handle, std::string &errstr); + + void disable_async_extract(); + + bool init(char *config, bool avoid_async); + void destroy(); + + virtual ss_plugin_type type() = 0; + + std::string get_last_error(); + + const std::string &name(); + const std::string &description(); + const std::string &contact(); + const version &plugin_version(); + const version &required_api_version(); + const filtercheck_field_info *fields(); + uint32_t nfields(); + + // Will either use the the async extractor interface or direct C calls to extract the values + bool extract_field(ss_plugin_event &evt, sinsp_plugin::ext_field &field); + +protected: + // Helper function to resolve symbols + static void* getsym(void* handle, const char* name, std::string &errstr); + + // Helper function to set a string from an allocated charbuf and free the charbuf. + static std::string str_from_alloc_charbuf(char *charbuf); + + // init() will call this to save the resulting state struct + virtual void set_plugin_state(ss_plugin_t *state) = 0; + virtual ss_plugin_t *plugin_state() = 0; + +private: + // Functions common to all derived plugin + // types. get_required_api_version/get_type are common but not + // included here as they are called in create_plugin() + typedef struct { + char* (*get_required_api_version)(); + ss_plugin_t* (*init)(char* config, int32_t* rc); + void (*destroy)(ss_plugin_t* s); + char* (*get_last_error)(ss_plugin_t* s); + char* (*get_name)(); + char* (*get_description)(); + char* (*get_contact)(); + char* (*get_version)(); + char* (*get_fields)(); + int32_t (*extract_fields)(ss_plugin_t *s, const ss_plugin_event *evt, uint32_t num_fields, ss_plugin_extract_field *fields); + int32_t (*register_async_extractor)(ss_plugin_t *s, async_extractor_info *info); + } common_plugin_info; + + std::string m_name; + std::string m_description; + std::string m_contact; + version m_plugin_version; + version m_required_api_version; + + // Allocated instead of vector to match how it will be held in filter_check_info + std::unique_ptr m_fields; + int32_t m_nfields; + + std::unique_ptr m_async_extractor; + + common_plugin_info m_plugin_info; +}; + +// Note that this doesn't have a next() method, as event generation is +// handled at the libscap level. +class sinsp_source_plugin : public sinsp_plugin +{ +public: + sinsp_source_plugin(); + virtual ~sinsp_source_plugin(); + + bool resolve_dylib_symbols(void *handle, std::string &errstr) override; + + ss_plugin_type type() override { return TYPE_SOURCE_PLUGIN; }; + uint32_t id(); + const std::string &event_source(); + + // For libscap that only works with struct of functions. + source_plugin_info *plugin_info(); + + // Note that embedding ss_instance_t in the object means that + // a plugin can only have one open active at a time. + bool open(char *params, int32_t &rc); + void close(); + std::string get_progress(uint32_t &progress_pct); + + std::string event_to_string(const uint8_t *data, uint32_t datalen); + +protected: + void set_plugin_state(ss_plugin_t *state) override; + virtual ss_plugin_t *plugin_state() override; + +private: + uint32_t m_id; + std::string m_event_source; + + source_plugin_info m_source_plugin_info; +}; + +class sinsp_extractor_plugin : public sinsp_plugin +{ +public: + sinsp_extractor_plugin(); + virtual ~sinsp_extractor_plugin(); + + bool resolve_dylib_symbols(void *handle, std::string &errstr) override; + + ss_plugin_type type() override { return TYPE_EXTRACTOR_PLUGIN; }; + + const std::set &extract_event_sources(); + + // Return true if the provided source is compatible with this + // extractor plugin, either because the extractor plugin does + // not name any extract sources, or if the provided source is + // in the set of extract sources. + bool source_compatible(const std::string &source); + +protected: + void set_plugin_state(ss_plugin_t *state) override; + virtual ss_plugin_t *plugin_state() override; + +private: + extractor_plugin_info m_extractor_plugin_info; + std::set m_extract_event_sources; +}; From 1b175074bfa782b5d9b456e1f30f553c37f67de2 Mon Sep 17 00:00:00 2001 From: Mark Stemm Date: Tue, 5 Oct 2021 13:46:46 -0700 Subject: [PATCH 026/148] Add new "plugin event" event type A plugin event has 2 fields: - plugin id - arbitrary event buffer, defined by the plugin This allows events created by plugins to live in capture files/be represented by sinsp_evt objects like all other events. Co-authored-by: Loris Degioanni Co-authored-by: Leonardo Grasso Signed-off-by: Mark Stemm --- driver/event_table.c | 4 +++- driver/ppm_events_public.h | 4 +++- 2 files changed, 6 insertions(+), 2 deletions(-) diff --git a/driver/event_table.c b/driver/event_table.c index 50567c27c..2926628fb 100644 --- a/driver/event_table.c +++ b/driver/event_table.c @@ -333,7 +333,9 @@ const struct ppm_event_info g_event_info[PPM_EVENT_MAX] = { /* PPME_SYSCALL_RENAMEAT2_E */{"renameat2", EC_FILE, EF_NONE, 0 }, /* PPME_SYSCALL_RENAMEAT2_X */{"renameat2", EC_FILE, EF_NONE, 6, {{"res", PT_ERRNO, PF_DEC}, {"olddirfd", PT_FD, PF_DEC}, {"oldpath", PT_FSRELPATH, PF_NA, DIRFD_PARAM(1)}, {"newdirfd", PT_FD, PF_DEC}, {"newpath", PT_FSRELPATH, PF_NA, DIRFD_PARAM(3)}, {"flags", PT_FLAGS32, PF_HEX, renameat2_flags} } }, /* PPME_SYSCALL_USERFAULTFD_E */{"userfaultfd", EC_FILE, EF_CREATES_FD | EF_MODIFIES_STATE, 0}, - /* PPME_SYSCALL_USERFAULTFD_X */{"userfaultfd", EC_FILE, EF_CREATES_FD | EF_MODIFIES_STATE, 2, {{"res", PT_ERRNO, PF_DEC}, {"flags", PT_FLAGS32, PF_HEX, file_flags} } } + /* PPME_SYSCALL_USERFAULTFD_X */{"userfaultfd", EC_FILE, EF_CREATES_FD | EF_MODIFIES_STATE, 2, {{"res", PT_ERRNO, PF_DEC}, {"flags", PT_FLAGS32, PF_HEX, file_flags} } }, + /* PPME_PLUGINEVENT_E */{"pluginevent", EC_OTHER, EF_NONE, 2, {{"plugin ID", PT_UINT32, PF_DEC}, {"event_data", PT_BYTEBUF, PF_NA} } }, + /* PPME_NA1 */{"pluginevent", EC_OTHER, EF_UNUSED, 0} /* NB: Starting from scap version 1.2, event types will no longer be changed when an event is modified, and the only kind of change permitted for pre-existent events is adding parameters. * New event types are allowed only for new syscalls or new internal events. * The number of parameters can be used to differentiate between event versions. diff --git a/driver/ppm_events_public.h b/driver/ppm_events_public.h index afea1266e..2329bcaf9 100644 --- a/driver/ppm_events_public.h +++ b/driver/ppm_events_public.h @@ -961,7 +961,9 @@ enum ppm_event_type { PPME_SYSCALL_RENAMEAT2_X = 319, PPME_SYSCALL_USERFAULTFD_E = 320, PPME_SYSCALL_USERFAULTFD_X = 321, - PPM_EVENT_MAX = 322 + PPME_PLUGINEVENT_E = 322, + PPME_PLUGINEVENT_X = 323, + PPM_EVENT_MAX = 324 }; /*@}*/ From 42921f2cd872c13065e17564c496b32572640f96 Mon Sep 17 00:00:00 2001 From: Mark Stemm Date: Tue, 5 Oct 2021 13:50:47 -0700 Subject: [PATCH 027/148] Use CMAKE_CURRENT_SOURCE_DIR for common This was required when libscap was included by other projects. Co-authored-by: Loris Degioanni Co-authored-by: Leonardo Grasso Signed-off-by: Mark Stemm --- userspace/libscap/CMakeLists.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/userspace/libscap/CMakeLists.txt b/userspace/libscap/CMakeLists.txt index f79b3523e..bc99ae1c2 100644 --- a/userspace/libscap/CMakeLists.txt +++ b/userspace/libscap/CMakeLists.txt @@ -13,7 +13,7 @@ # See the License for the specific language governing permissions and # limitations under the License. # -include_directories(../../common) +include_directories("${CMAKE_CURRENT_SOURCE_DIR}/../../common") option(USE_BUNDLED_DEPS "Enable bundled dependencies instead of using the system ones" ON) From f557172cd77717a0268a8358f6cc4b57482431a0 Mon Sep 17 00:00:00 2001 From: Mark Stemm Date: Tue, 5 Oct 2021 13:55:32 -0700 Subject: [PATCH 028/148] libscap support for source plugins Add support for opening plugins and sourcing events from plugins in libscap: - The structs in plugin_info represent the interface to a plugin dynamic library. Required enums/consts used for the plugin interface are defined here. The source_plugin_info/extractor_plugin_info structs contain the resolved functions. - scap_open_plugin_int() handles the details of opening a plugin. It takes a source_plugin_info struct and calls the plugin's open() function. - scap_next_plugin() calls the plugin's next() method to return events. Capture file handling also has some changes, with the introduction of plugin sources: - When writing capture files from plugin events, the resulting capture file does not have any of the following: - fd list - process list - machine info - interface list - user list - When reading capture files, libscap does not mandate any of the above blocks, as they could have been written by a libscap reading plugin events. - When reading capture files, if a section header block is found in the middle of the file, instead of returning SCAP_UNEXPECTED_BLOCK, read the section header, make no changes, and return SCAP_TIMEOUT. This allows a capture file to contain a section header block in the middle, which supports use cases involving concatenating .scap files from plugins into a single file. Co-authored-by: Loris Degioanni Co-authored-by: Leonardo Grasso Signed-off-by: Mark Stemm --- userspace/libscap/plugin_info.h | 526 ++++++++++++++++++++++++++++++ userspace/libscap/scap-int.h | 22 ++ userspace/libscap/scap.c | 261 ++++++++++++++- userspace/libscap/scap.h | 8 + userspace/libscap/scap_savefile.c | 158 ++++++--- userspace/libscap/scap_savefile.h | 16 +- 6 files changed, 932 insertions(+), 59 deletions(-) create mode 100644 userspace/libscap/plugin_info.h diff --git a/userspace/libscap/plugin_info.h b/userspace/libscap/plugin_info.h new file mode 100644 index 000000000..57dbf0a74 --- /dev/null +++ b/userspace/libscap/plugin_info.h @@ -0,0 +1,526 @@ +/* +Copyright (C) 2021 The Falco Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. + +*/ + +#pragma once + +#include +#include + +// +// This file contains the prototype and type definitions of sinsp/scap plugins +// + +// +// API versions of this plugin engine +// +#define PLUGIN_API_VERSION_MAJOR 0 +#define PLUGIN_API_VERSION_MINOR 1 +#define PLUGIN_API_VERSION_PATCH 0 + +// +// There are two plugin types: source plugins and extractor plugins. +// +// Source plugins implement a new sinsp/scap event source and have the +// ability to provide events to the event loop. Optionally, they can +// extract fields from events so they can be displayed/used in +// filters. +// +// Extractor plugins do not provide events, but have the ability to +// extract fields from events created by other plugins. A good example +// of an extractor plugin is a json extractor, which can extract +// information from any json payload, regardless of where the payloads +// come from. +// +typedef enum ss_plugin_type +{ + TYPE_SOURCE_PLUGIN = 1, + TYPE_EXTRACTOR_PLUGIN = 2 +}ss_plugin_type; + +// The noncontinguous numbers are to maintain equality with underlying +// falcosecurity libs types. +typedef enum ss_plugin_field_type +{ + FTYPE_UINT64 = 8, + FTYPE_STRING = 9 +}ss_plugin_field_type; + +// Values to return from init() / open() / next() / next_batch() / +// extract_fields() / register_async_extractor(). Note that these +// values map exactly to the corresponding SCAP_XXX values in scap.h, +// and should be kept in sync. +#define SS_PLUGIN_SUCCESS 0 +#define SS_PLUGIN_FAILURE 1 +#define SS_PLUGIN_TIMEOUT -1 +#define SS_PLUGIN_ILLEGAL_INPUT 3 +#define SS_PLUGIN_NOTFOUND 4 +#define SS_PLUGIN_INPUT_TOO_SMALL 5 +#define SS_PLUGIN_EOF 6 +#define SS_PLUGIN_UNEXPECTED_BLOCK 7 +#define SS_PLUGIN_VERSION_MISMATCH 8 +#define SS_PLUGIN_NOT_SUPPORTED 9 + + +// This struct represents an event returned by the plugin, and is used +// below in next()/next_batch(). +// - evtnum: incremented for each event returned. Might not be contiguous. +// - data: pointer to a memory buffer pointer. The plugin will set it +// to point to the memory containing the next event. Once returned, +// the memory is owned by the plugin framework and will be freed via +// a call to free(). +// - datalen: pointer to a 32bit integer. The plugin will set it the size of the +// buffer pointed by data. +// - ts: the event timestamp, in nanoseconds since the epoch. +// Can be (uint64_t)-1, in which case the engine will automatically +// fill the event time with the current time. +// +// Note: event numbers are assigned by the plugin +// framework. Therefore, there isn't any need to fill in evtnum when +// returning an event via plugin_next/plugin_next_batch. It will be ignored. +typedef struct ss_plugin_event +{ + uint64_t evtnum; + uint8_t *data; + uint32_t datalen; + uint64_t ts; +} ss_plugin_event; + +// Used in extract_fields functions below to receive a field/arg +// pair and return an extracted value. +// field: the field name. +// arg: the field argument, if an argument has been specified +// for the field, otherwise it's NULL. +// For example: +// * if the field specified by the user is foo.bar[pippo], arg will be the +// string "pippo" +// * if the field specified by the user is foo.bar, arg will be NULL +// ftype: the type of the field. Could be derived from the field name alone, +// but including here can prevent a second lookup of field names. +// The following should be filled in by the extraction function: +// - field_present: set to true if the event has a meaningful +// extracted value for the provided field, false otherwise +// - res_str: if the corresponding field was type==string, this should be +// filled in with the string value. The string should be allocated by +// the plugin using malloc() and will be free()d by the plugin framework. +// - res_u64: if the corresponding field was type==uint64, this should be +// filled in with the uint64 value. + +typedef struct ss_plugin_extract_field +{ + const char *field; + const char *arg; + uint32_t ftype; + + bool field_present; + char *res_str; + uint64_t res_u64; +} ss_plugin_extract_field; + +// Used by the async extraction interface +typedef bool (*cb_wait_t)(void* wait_ctx); + +typedef struct async_extractor_info +{ + // Pointer as this allows swapping out events from other + // structs. + const ss_plugin_event *evt; + ss_plugin_extract_field *field; + int32_t rc; + cb_wait_t cb_wait; + void* wait_ctx; +} async_extractor_info; + +// +// This is the opaque pointer to the state of a plugin. +// It points to any data that might be needed plugin-wise. It is +// allocated by init() and must be destroyed by destroy(). +// It is defined as void because the engine doesn't care what it is +// and it treats is as opaque. +// +typedef void ss_plugin_t; + +// +// This is the opaque pointer to the state of an open instance of the source +// plugin. +// It points to any data that is needed while a capture is running. It is +// allocated by open() and must be destroyed by close(). +// It is defined as void because the engine doesn't care what it is +// and it treats is as opaque. +// +typedef void ss_instance_t; + +// +// The structs below define the functions and arguments for source and +// exctractor plugins. The structs are used by the plugin framework to +// load and interface with plugins. +// +// From the perspective of the plugin, each function below should be +// exported from the dynamic library as a C calling convention +// function, adding a prefix "plugin_" to the function name +// (e.g. plugin_get_required_api_version, plugin_init, etc.) +// +// NOTE: For all functions below that return a char */struct *, the memory +// pointed to by the char */struct * must be allocated by the plugin using +// malloc() and should be freed by the caller using free(). +// + +// +// Interface for a sinsp/scap source plugin. +// +typedef struct +{ + // + // Return the version of the plugin API used by this plugin. + // Required: yes + // Return value: the API version string, in the following format: + // "..", e.g. "1.2.3". + // NOTE: to ensure correct interoperability between the engine and the plugins, + // we use a semver approach. Plugins are required to specify the version + // of the API they run against, and the engine will take care of checking + // and enforcing compatibility. + // + char* (*get_required_api_version)(); + // + // Return the plugin type. + // Required: yes + // Should return TYPE_SOURCE_PLUGIN. It still makes sense to + // have a function get_type() as the plugin interface will + // often dlsym() functions from shared libraries, and can't + // inspect any C struct type. + // + uint32_t (*get_type)(); + // + // Initialize the plugin and, if needed, allocate its state. + // Required: yes + // Arguments: + // - config: a string with the plugin configuration. The format of the + // string is chosen by the plugin itself. + // - rc: pointer to an integer that will contain the initialization result, + // as a SS_PLUGIN_* value (e.g. SS_PLUGIN_SUCCESS=0, SS_PLUGIN_FAILURE=1) + // Return value: pointer to the plugin state that will be treated as opaque + // by the engine and passed to the other plugin functions. + // If rc is SS_PLUGIN_FAILURE, this function should return NULL. + // + ss_plugin_t* (*init)(char* config, int32_t* rc); + // + // Destroy the plugin and, if plugin state was allocated, free it. + // Required: yes + // + void (*destroy)(ss_plugin_t* s); + // + // Return a string with the error that was last generated by + // the plugin. + // Required: yes + // + // In cases where any other api function returns an error, the + // plugin should be prepared to return a human-readable error + // string with more context for the error. The plugin manager + // calls get_last_error() to access that string. + // + char* (*get_last_error)(ss_plugin_t* s); + // + // Return the unique ID of the plugin. + // Required: yes + // EVERY SOURCE PLUGIN (see get_type()) MUST OBTAIN AN OFFICIAL ID FROM THE + // FALCOSECURITY ORGANIZATION, OTHERWISE IT WON'T PROPERLY COEXIST WITH OTHER PLUGINS. + // + uint32_t (*get_id)(); + // + // Return the name of the plugin, which will be printed when displaying + // information about the plugin. + // Required: yes + // + char* (*get_name)(); + // + // Return the descriptions of the plugin, which will be printed when displaying + // information about the plugin or its events. + // Required: yes + // + char* (*get_description)(); + // + // Return a string containing contact info (url, email, twitter, etc) for + // the plugin authors. + // Required: yes + // + char* (*get_contact)(); + // + // Return the version of this plugin itself + // Required: yes + // Return value: a string with a version identifier, in the following format: + // "..", e.g. "1.2.3". + // This differs from the api version in that this versions the + // plugin itself, as compared to the plugin interface. When + // reading capture files, the major version of the plugin that + // generated events must match the major version of the plugin + // used to read events. + // + char* (*get_version)(); + // + // Return a string describing the events generated by this source plugin. + // Required: yes + // Example event sources would be strings like "syscall", + // "k8s_audit", etc. The source can be used by extractor + // plugins to filter the events they receive. + // + char* (*get_event_source)(); + // + // Return the list of extractor fields exported by this plugin. Extractor + // fields can be used in Falco rule conditions and sysdig filters. + // Required: no + // Return value: a string with the list of fields encoded as a json + // array. + // Each field entry is a json object with the following properties: + // "type": one of "string", "uint64" + // "name": a string with a name for the field + // "argRequired: (optional) If present and set to true, notes + // that the field requires an argument e.g. field[arg]. + // "display": (optional) If present, a string that will be used to + // display the field instead of the name. Used in tools + // like wireshark. + // "desc": a string with a description of the field + // Example return value: + // [ + // {"type": "string", "name": "field1", "argRequired": true, "desc": "Describing field 1"}, + // {"type": "uint64", "name": "field2", "desc": "Describing field 2"} + // ] + char* (*get_fields)(); + // + // Open the source and start a capture (e.g. stream of events) + // Required: yes + // Arguments: + // - s: the plugin state returned by init() + // - params: the open parameters, as a string. The format is defined by the plugin + // itsef + // - rc: pointer to an integer that will contain the open result, as a SS_PLUGIN_* value + // (e.g. SS_PLUGIN_SUCCESS=0, SS_PLUGIN_FAILURE=1) + // Return value: a pointer to the open context that will be passed to next(), + // close(), event_to_string() and extract_fields. + // + ss_instance_t* (*open)(ss_plugin_t* s, char* params, int32_t* rc); + // + // Close a capture. + // Required: yes + // Arguments: + // - s: the plugin context, returned by init(). Can be NULL. + // - h: the capture context, returned by open(). Can be NULL. + // + void (*close)(ss_plugin_t* s, ss_instance_t* h); + // + // Return the next event. + // Required: yes + // Arguments: + // - s: the plugin context, returned by init(). Can be NULL. + // - h: the capture context, returned by open(). Can be NULL. + // + // - evt: pointer to a ss_plugin_event pointer. The plugin should + // allocate a ss_plugin_event struct using malloc(), as well as + // allocate the data buffer within the ss_plugin_event struct. + // Both the struct and data buffer are owned by the plugin framework + // and will free them using free(). + // + // Return value: the status of the operation (e.g. SS_PLUGIN_SUCCESS=0, SS_PLUGIN_FAILURE=1, + // SS_PLUGIN_TIMEOUT=-1) + // + int32_t (*next)(ss_plugin_t* s, ss_instance_t* h, ss_plugin_event **evt); + // + // Return the read progress. + // Required: no + // Arguments: + // - progress_pct: the read progress, as a number between 0 (no data has been read) + // and 10000 (100% of the data has been read). This encoding allows the engine to + // print progress decimals without requiring to deal with floating point numbers + // (which could cause incompatibility problems with some languages). + // Return value: a string representation of the read + // progress. This might include the progress percentage + // combined with additional context added by the plugin. If + // NULL, progress_pct should be used. + // NOTE: reporting progress is optional and in some case could be impossible. However, + // when possible, it's recommended as it provides valuable information to the + // user. + // + char* (*get_progress)(ss_plugin_t* s, ss_instance_t* h, uint32_t* progress_pct); + // + // Return a text representation of an event generated by this source plugin. + // Required: yes + // Arguments: + // - data: the buffer from an event produced by next(). + // - datalen: the length of the buffer from an event produced by next(). + // Return value: the text representation of the event. This is used, for example, + // by sysdig to print a line for the given event. + // + char *(*event_to_string)(ss_plugin_t *s, const uint8_t *data, uint32_t datalen); + // + // Extract one or more a filter field values from an event. + // Required: no + // Arguments: + // - evt: an event struct returned by a call to next()/batch_next(). + // - num_fields: the length of the fields array. + // - fields: an array of ss_plugin_extract_field structs. Each element contains + // a single field + optional arg, and the corresponding extracted value should + // be in the same struct. + // Return value: An integer with values SS_PLUGIN_SUCCESS or SS_PLUGIN_FAILURE. + // + int32_t (*extract_fields)(ss_plugin_t *s, const ss_plugin_event *evt, uint32_t num_fields, ss_plugin_extract_field *fields); + // + // This is an optional, internal, function used to speed up event capture by + // batching the calls to next(). + // On success: + // - nevts will be filled in with the number of events. + // - evts: pointer to an ss_plugin_event pointer. The plugin should + // allocate an array of contiguous ss_plugin_event structs using malloc(), + // as well as allocate each data buffer within each ss_plugin_event + // struct using malloc(). Both the array of structs and each data buffer are + // owned by the plugin framework and will free them using free(). + // Required: no + // + int32_t (*next_batch)(ss_plugin_t* s, ss_instance_t* h, uint32_t *nevts, ss_plugin_event **evts); + // + // This is an optional, internal, function used to speed up value extraction + // Required: no + // + int32_t (*register_async_extractor)(ss_plugin_t *s, async_extractor_info *info); + + // + // The following members are PRIVATE for the engine and should not be touched. + // + ss_plugin_t* state; + ss_instance_t* handle; + uint32_t id; + char *name; +} source_plugin_info; + +// +// Interface for a sinsp/scap extractor plugin +// +typedef struct +{ + // + // Return the version of the plugin API used by this plugin. + // Required: yes + // Return value: the API version string, in the following format: + // "..", e.g. "1.2.3". + // NOTE: to ensure correct interoperability between the engine and the plugins, + // we use a semver approach. Plugins are required to specify the version + // of the API they run against, and the engine will take care of checking + // and enforcing compatibility. + // + char* (*get_required_api_version)(); + // + // Return the plugin type. + // Required: yes + // Should return TYPE_EXTRACTOR_PLUGIN. It still makes sense to + // have a function get_type() as the plugin interface will + // often dlsym() functions from shared libraries, and can't + // inspect any C struct type. + // + uint32_t (*get_type)(); + // + // Initialize the plugin and, if needed, allocate its state. + // Required: yes + // Arguments: + // - config: a string with the plugin configuration. The format of the + // string is chosen by the plugin itself. + // - rc: pointer to an integer that will contain the initialization result, + // as a SS_PLUGIN_* value (e.g. SS_PLUGIN_SUCCESS=0, SS_PLUGIN_FAILURE=1) + // Return value: pointer to the plugin state that will be treated as opaque + // by the engine and passed to the other plugin functions. + // + ss_plugin_t* (*init)(char* config, int32_t* rc); + // + // Destroy the plugin and, if plugin state was allocated, free it. + // Required: yes + // + void (*destroy)(ss_plugin_t* s); + // + // Return a string with the error that was last generated by + // the plugin. + // Required: yes + // + // In cases where any other api function returns an error, the + // plugin should be prepared to return a human-readable error + // string with more context for the error. The plugin manager + // calls get_last_error() to access that string. + // + char* (*get_last_error)(ss_plugin_t* s); + // + // Return the name of the plugin, which will be printed when displaying + // information about the plugin. + // Required: yes + // + char* (*get_name)(); + // + // Return the descriptions of the plugin, which will be printed when displaying + // information about the plugin or its events. + // Required: yes + // + char* (*get_description)(); + // + // Return a string containing contact info (url, email, twitter, etc) for + // the plugin author. + // Required: yes + // + char* (*get_contact)(); + // + // Return the version of this plugin itself + // Required: yes + // Return value: a string with a version identifier, in the following format: + // "..", e.g. "1.2.3". + // This differs from the api version in that this versions the + // plugin itself, as compared to the plugin interface. When + // reading capture files, the major version of the plugin that + // generated events must match the major version of the plugin + // used to read events. + // + char* (*get_version)(); + // + // Return a string describing the event sources that this + // extractor plugin can consume. + // Required: no + // Return value: a json array of strings containing event + // sources returned by a source plugin's get_event_source() + // function. + // This function is optional--if NULL then the exctractor + // plugin will receive every event. + // + char* (*get_extract_event_sources)(); + // + // Return the list of extractor fields exported by this plugin. Extractor + // fields can be used in Falco rules and sysdig filters. + // Required: yes + // Return value: a string with the list of fields encoded as a json + // array. + // + char* (*get_fields)(); + + // + // Extract one or more a filter field values from an event. + // Required: no + // Arguments: + // - evt: an event struct returned by a call to next()/batch_next(). + // - num_fields: the length of the fields array. + // - fields: an array of ss_plugin_extract_field structs. Each element contains + // a single field + optional arg, and the corresponding extracted value should + // be in the same struct. + // Return value: An integer with values SS_PLUGIN_SUCCESS or SS_PLUGIN_FAILURE. + // + int32_t (*extract_fields)(ss_plugin_t *s, const ss_plugin_event *evt, uint32_t num_fields, ss_plugin_extract_field *fields); + + // + // The following members are PRIVATE for the engine and should not be touched. + // + ss_plugin_t* state; +} extractor_plugin_info; diff --git a/userspace/libscap/scap-int.h b/userspace/libscap/scap-int.h index 97a835530..fb712b30a 100644 --- a/userspace/libscap/scap-int.h +++ b/userspace/libscap/scap-int.h @@ -174,6 +174,28 @@ struct scap // The number of events that were skipped due to the comm // matching an entry in m_suppressed_comms. uint64_t m_num_suppressed_evts; + + // + // Plugin-related state + // + source_plugin_info* m_input_plugin; + uint8_t* m_input_plugin_evt_storage; + uint32_t m_input_plugin_evt_storage_len; + + // The number of items held in batch_evts + uint32_t m_input_plugin_batch_nevts; + + // A set of events returned from next_batch. The array is + // allocated and must be free()d when done. + ss_plugin_event* m_input_plugin_batch_evts; + + // The current position into the above arrays (0-indexed), + // reflecting how many of the above items have been returned + // via a call to next(). + uint32_t m_input_plugin_batch_idx; + + // The return value from the last call to batch_next(). + int32_t m_input_plugin_last_batch_res; }; typedef enum ppm_dumper_type diff --git a/userspace/libscap/scap.c b/userspace/libscap/scap.c index 149392ec1..dfe3b6380 100644 --- a/userspace/libscap/scap.c +++ b/userspace/libscap/scap.c @@ -86,6 +86,20 @@ static int32_t copy_comms(scap_t *handle, const char **suppressed_comms) return SCAP_SUCCESS; } +static void scap_free_plugin_batch_state(scap_t* handle) +{ + for(uint32_t i = 0; i < handle->m_input_plugin_batch_nevts; i++) + { + free(handle->m_input_plugin_batch_evts[i].data); + } + + handle->m_input_plugin_batch_nevts = 0; + handle->m_input_plugin_batch_idx = 0; + + free(handle->m_input_plugin_batch_evts); + handle->m_input_plugin_batch_evts = NULL; +} + #if !defined(HAS_CAPTURE) || defined(CYGWING_AGENT) || defined(_WIN32) scap_t* scap_open_live_int(char *error, int32_t *rc, proc_entry_callback proc_callback, @@ -638,10 +652,11 @@ scap_t* scap_open_udig_int(char *error, int32_t *rc, // error[0] = '\0'; snprintf(filename, sizeof(filename), "%s/proc", scap_get_host_root()); - if((*rc = scap_proc_scan_proc_dir(handle, filename, error)) != SCAP_SUCCESS) + char procerr[SCAP_LASTERR_SIZE]; + if((*rc = scap_proc_scan_proc_dir(handle, filename, procerr)) != SCAP_SUCCESS) { scap_close(handle); - snprintf(error, SCAP_LASTERR_SIZE, "%s", error); + snprintf(error, SCAP_LASTERR_SIZE, "%s", procerr); return NULL; } @@ -918,6 +933,83 @@ scap_t* scap_open_nodriver_int(char *error, int32_t *rc, #endif // HAS_CAPTURE } +scap_t* scap_open_plugin_int(char *error, int32_t *rc, source_plugin_info* input_plugin, char* input_plugin_params) +{ + scap_t* handle = NULL; + + // + // Allocate the handle + // + handle = (scap_t*)malloc(sizeof(scap_t)); + if(!handle) + { + snprintf(error, SCAP_LASTERR_SIZE, "error allocating the scap_t structure"); + *rc = SCAP_FAILURE; + return NULL; + } + + // + // Preliminary initializations + // + memset(handle, 0, sizeof(scap_t)); + handle->m_mode = SCAP_MODE_PLUGIN; + + // + // Extract machine information + // + handle->m_proc_callback = NULL; + handle->m_proc_callback_context = NULL; +#ifdef _WIN32 + handle->m_machine_info.num_cpus = 0; + handle->m_machine_info.memory_size_bytes = 0; +#else + handle->m_machine_info.num_cpus = sysconf(_SC_NPROCESSORS_ONLN); + handle->m_machine_info.memory_size_bytes = (uint64_t)sysconf(_SC_PHYS_PAGES) * sysconf(_SC_PAGESIZE); +#endif + gethostname(handle->m_machine_info.hostname, sizeof(handle->m_machine_info.hostname) / sizeof(handle->m_machine_info.hostname[0])); + handle->m_machine_info.reserved1 = 0; + handle->m_machine_info.reserved2 = 0; + handle->m_machine_info.reserved3 = 0; + handle->m_machine_info.reserved4 = 0; + handle->m_driver_procinfo = NULL; + handle->m_fd_lookup_limit = SCAP_NODRIVER_MAX_FD_LOOKUP; // fd lookup is limited here because is very expensive + handle->m_fake_kernel_proc.tid = -1; + handle->m_fake_kernel_proc.pid = -1; + handle->m_fake_kernel_proc.flags = 0; + snprintf(handle->m_fake_kernel_proc.comm, SCAP_MAX_PATH_SIZE, "kernel"); + snprintf(handle->m_fake_kernel_proc.exe, SCAP_MAX_PATH_SIZE, "kernel"); + handle->m_fake_kernel_proc.args[0] = 0; + handle->refresh_proc_table_when_saving = true; + + handle->m_input_plugin = input_plugin; + handle->m_input_plugin->name = handle->m_input_plugin->get_name(); + handle->m_input_plugin->id = handle->m_input_plugin->get_id(); + + // Set the rc to SCAP_FAILURE now, so in the unlikely event + // that a plugin doesn't not actually set a rc, that it gets + // treated as a failure. + *rc = SCAP_FAILURE; + + handle->m_input_plugin->handle = handle->m_input_plugin->open(handle->m_input_plugin->state, + input_plugin_params, + rc); + handle->m_input_plugin_batch_nevts = 0; + handle->m_input_plugin_batch_evts = NULL; + handle->m_input_plugin_batch_idx = 0; + handle->m_input_plugin_last_batch_res = SCAP_SUCCESS; + + if(*rc != SCAP_SUCCESS) + { + char *errstr = handle->m_input_plugin->get_last_error(handle->m_input_plugin->state); + snprintf(error, SCAP_LASTERR_SIZE, "%s", errstr); + free(errstr); + scap_close(handle); + return NULL; + } + + return handle; +} + scap_t* scap_open(scap_open_args args, char *error, int32_t *rc) { switch(args.mode) @@ -972,7 +1064,7 @@ scap_t* scap_open(scap_open_args args, char *error, int32_t *rc) args.suppressed_comms); } #else - snprintf(error, SCAP_LASTERR_SIZE, "scap_open: live mode currently not supported on windows. Use nodriver mode instead."); + snprintf(error, SCAP_LASTERR_SIZE, "scap_open: live mode currently not supported on Windows."); *rc = SCAP_NOT_SUPPORTED; return NULL; #endif @@ -980,6 +1072,8 @@ scap_t* scap_open(scap_open_args args, char *error, int32_t *rc) return scap_open_nodriver_int(error, rc, args.proc_callback, args.proc_callback_context, args.import_users); + case SCAP_MODE_PLUGIN: + return scap_open_plugin_int(error, rc, args.input_plugin, args.input_plugin_params); case SCAP_MODE_NONE: // error break; @@ -1079,6 +1173,13 @@ void scap_close(scap_t* handle) } #endif // HAS_CAPTURE } + else if(handle->m_mode == SCAP_MODE_PLUGIN) + { + handle->m_input_plugin->close(handle->m_input_plugin->state, handle->m_input_plugin->handle); + scap_free_plugin_batch_state(handle); + // name was allocated + free(handle->m_input_plugin->name); + } #if CYGWING_AGENT || _WIN32 if(handle->m_whh != NULL) @@ -1581,6 +1682,157 @@ static int32_t scap_next_nodriver(scap_t* handle, OUT scap_evt** pevent, OUT uin } #endif // _WIN32 +static inline uint64_t get_timestamp_ns() +{ + uint64_t ts; + +#ifdef _WIN32 + FILETIME ft; + static const uint64_t EPOCH = ((uint64_t) 116444736000000000ULL); + + GetSystemTimePreciseAsFileTime(&ft); + + uint64_t ftl = (((uint64_t)ft.dwHighDateTime) << 32) + ft.dwLowDateTime; + ftl -= EPOCH; + + ts = ftl * 100; +#else + struct timeval tv; + gettimeofday(&tv, NULL); + ts = tv.tv_sec * (uint64_t) 1000000000 + tv.tv_usec * 1000; +#endif + + return ts; +} + +static int32_t scap_next_plugin(scap_t* handle, OUT scap_evt** pevent, OUT uint16_t* pcpuid) +{ + ss_plugin_event *plugin_evt; + int32_t res = SCAP_FAILURE; + bool should_free_plugin_evt = false; + + if(handle->m_input_plugin->next_batch != NULL) + { + if(handle->m_input_plugin_batch_idx >= handle->m_input_plugin_batch_nevts) + { + scap_free_plugin_batch_state(handle); + + if(handle->m_input_plugin_last_batch_res != SCAP_SUCCESS) + { + if(handle->m_input_plugin_last_batch_res != SCAP_TIMEOUT && handle->m_input_plugin_last_batch_res != SCAP_EOF) + { + char *errstr = handle->m_input_plugin->get_last_error(handle->m_input_plugin->state); + snprintf(handle->m_lasterr, SCAP_LASTERR_SIZE, "%s", errstr); + free(errstr); + } + int32_t tres = handle->m_input_plugin_last_batch_res; + handle->m_input_plugin_last_batch_res = SCAP_SUCCESS; + return tres; + } + + handle->m_input_plugin_last_batch_res = handle->m_input_plugin->next_batch(handle->m_input_plugin->state, + handle->m_input_plugin->handle, + &(handle->m_input_plugin_batch_nevts), + &(handle->m_input_plugin_batch_evts)); + + if(handle->m_input_plugin_batch_nevts == 0) + { + if(handle->m_input_plugin_last_batch_res == SCAP_SUCCESS) + { + snprintf(handle->m_lasterr, SCAP_LASTERR_SIZE, "unexpected 0 size event returned by plugin %s", handle->m_input_plugin->name); + ASSERT(false); + return SCAP_FAILURE; + } + else + { + if(handle->m_input_plugin_last_batch_res != SCAP_TIMEOUT && handle->m_input_plugin_last_batch_res != SCAP_EOF) + { + char *errstr = handle->m_input_plugin->get_last_error(handle->m_input_plugin->state); + snprintf(handle->m_lasterr, SCAP_LASTERR_SIZE, "%s", errstr); + free(errstr); + } + return handle->m_input_plugin_last_batch_res; + } + } + + handle->m_input_plugin_batch_idx = 0; + } + + uint32_t pos = handle->m_input_plugin_batch_idx; + + plugin_evt = &(handle->m_input_plugin_batch_evts[pos]); + + handle->m_input_plugin_batch_idx++; + + res = SCAP_SUCCESS; + } + else + { + should_free_plugin_evt = true; + + res = handle->m_input_plugin->next(handle->m_input_plugin->state, + handle->m_input_plugin->handle, &plugin_evt); + if(res != SCAP_SUCCESS) + { + if(res != SCAP_TIMEOUT && res != SCAP_EOF) + { + char *errstr = handle->m_input_plugin->get_last_error(handle->m_input_plugin->state); + snprintf(handle->m_lasterr, SCAP_LASTERR_SIZE, "%s", errstr); + free(errstr); + } + return res; + } + + res = SCAP_SUCCESS; + } + + uint32_t reqsize = sizeof(scap_evt) + 2 + 4 + 2 + plugin_evt->datalen; + if(handle->m_input_plugin_evt_storage_len < reqsize) + { + handle->m_input_plugin_evt_storage = (uint8_t*)realloc(handle->m_input_plugin_evt_storage, reqsize); + handle->m_input_plugin_evt_storage_len = reqsize; + } + + scap_evt* evt = (scap_evt*)handle->m_input_plugin_evt_storage; + evt->len = reqsize; + evt->tid = -1; + evt->type = PPME_PLUGINEVENT_E; + evt->nparams = 2; + + uint8_t* buf = handle->m_input_plugin_evt_storage + sizeof(scap_evt); + + const uint16_t four = 4; + memcpy(buf, &four, sizeof(four)); + buf += sizeof(four); + + memcpy(buf, &(plugin_evt->datalen), sizeof(plugin_evt->datalen)); + buf += sizeof(plugin_evt->datalen); + + memcpy(buf, &(handle->m_input_plugin->id), sizeof(handle->m_input_plugin->id)); + buf += sizeof(handle->m_input_plugin->id); + + memcpy(buf, plugin_evt->data, plugin_evt->datalen); + + if(should_free_plugin_evt) + { + free(plugin_evt->data); + plugin_evt->data = NULL; + free(plugin_evt); + } + + if(plugin_evt->ts != UINT64_MAX) + { + evt->ts = plugin_evt->ts; + } + else + { + evt->ts = get_timestamp_ns(); + } + + *pevent = evt; + return res; +} + uint64_t scap_max_buf_used(scap_t* handle) { #if defined(HAS_CAPTURE) && !defined(CYGWING_AGENT) @@ -1623,6 +1875,9 @@ int32_t scap_next(scap_t* handle, OUT scap_evt** pevent, OUT uint16_t* pcpuid) res = scap_next_nodriver(handle, pevent, pcpuid); break; #endif + case SCAP_MODE_PLUGIN: + res = scap_next_plugin(handle, pevent, pcpuid); + break; case SCAP_MODE_NONE: res = SCAP_FAILURE; } diff --git a/userspace/libscap/scap.h b/userspace/libscap/scap.h index aa02a9a2b..ed7626add 100644 --- a/userspace/libscap/scap.h +++ b/userspace/libscap/scap.h @@ -65,6 +65,8 @@ struct iovec; #define MAP_FAILED (void*)-1 #endif +#include "plugin_info.h" + // // Return types // @@ -288,6 +290,10 @@ typedef enum { * returned. */ SCAP_MODE_NODRIVER, + /*! + * Do not read system call data. Events come from the configured input plugin. + */ + SCAP_MODE_PLUGIN, } scap_mode_t; typedef struct scap_open_args @@ -305,6 +311,8 @@ typedef struct scap_open_args // You can provide additional comm // values via scap_suppress_events_comm(). bool udig; ///< If true, UDIG will be used for event capture. Otherwise, the kernel driver will be used. + source_plugin_info* input_plugin; ///< use this to configure a source plugin that will produce the events for this capture + char* input_plugin_params; ///< optional parameters string for the source plugin pointed by src_plugin }scap_open_args; diff --git a/userspace/libscap/scap_savefile.c b/userspace/libscap/scap_savefile.c index 394a67b4b..c3505c331 100755 --- a/userspace/libscap/scap_savefile.c +++ b/userspace/libscap/scap_savefile.c @@ -236,6 +236,14 @@ static int32_t scap_write_fdlist(scap_t *handle, scap_dumper_t *d) struct scap_threadinfo *ttinfo; int32_t res; + // + // No fd list on disk if the source is a plugin + // + if(handle->m_mode == SCAP_MODE_PLUGIN) + { + return SCAP_SUCCESS; + } + HASH_ITER(hh, handle->m_proclist, tinfo, ttinfo) { if(!tinfo->filtered_out) @@ -420,6 +428,14 @@ static int32_t scap_write_proclist(scap_t *handle, scap_dumper_t *d) struct scap_threadinfo *tinfo; struct scap_threadinfo *ttinfo; + // + // No process list on disk if the source is a plugin + // + if(handle->m_mode == SCAP_MODE_PLUGIN) + { + return SCAP_SUCCESS; + } + uint32_t* lengths = calloc(HASH_COUNT(handle->m_proclist), sizeof(uint32_t)); if(lengths == NULL) { @@ -507,6 +523,14 @@ static int32_t scap_write_machine_info(scap_t *handle, scap_dumper_t *d) block_header bh; uint32_t bt; + // + // No machine info on disk if the source is a plugin + // + if(handle->m_mode == SCAP_MODE_PLUGIN) + { + return SCAP_SUCCESS; + } + // // Write the section header // @@ -537,14 +561,23 @@ static int32_t scap_write_iflist(scap_t *handle, scap_dumper_t* d) uint32_t totlen = 0; uint32_t j; + // + // No interface list on disk if the source is a plugin + // + if(handle->m_mode == SCAP_MODE_PLUGIN) + { + return SCAP_SUCCESS; + } + // // Get the interface list // if(handle->m_addrlist == NULL) { - ASSERT(false); - snprintf(handle->m_lasterr, SCAP_LASTERR_SIZE, "error writing to trace file: interface list missing"); - return SCAP_FAILURE; + // + // This can happen when the event source is a capture that was generated by a plugin, no big deal + // + return SCAP_SUCCESS; } // @@ -646,14 +679,23 @@ static int32_t scap_write_userlist(scap_t *handle, scap_dumper_t* d) uint8_t type; uint32_t totlen = 0; + // + // No user list on disk if the source is a plugin + // + if(handle->m_mode == SCAP_MODE_PLUGIN) + { + return SCAP_SUCCESS; + } + // // Make sure we have a user list interface list // if(handle->m_userlist == NULL) { - ASSERT(false); - snprintf(handle->m_lasterr, SCAP_LASTERR_SIZE, "error writing to trace file: user list missing"); - return SCAP_FAILURE; + // + // This can happen when the event source is a capture that was generated by a plugin, no big deal + // + return SCAP_SUCCESS; } uint32_t* lengths = calloc(handle->m_userlist->nusers + handle->m_userlist->ngroups, sizeof(uint32_t)); @@ -2482,6 +2524,37 @@ static int32_t scap_read_fdlist(scap_t *handle, gzFile f, uint32_t block_length, return SCAP_SUCCESS; } +int32_t scap_read_section_header(scap_t *handle, gzFile f) +{ + section_header_block sh; + uint32_t bt; + + // + // Read the section header block + // + if(gzread(f, &sh, sizeof(sh)) != sizeof(sh) || + gzread(f, &bt, sizeof(bt)) != sizeof(bt)) + { + snprintf(handle->m_lasterr, SCAP_LASTERR_SIZE, "error reading from file (1)"); + return SCAP_FAILURE; + } + + if(sh.byte_order_magic != 0x1a2b3c4d) + { + snprintf(handle->m_lasterr, SCAP_LASTERR_SIZE, "invalid magic number"); + return SCAP_FAILURE; + } + + if(sh.major_version > CURRENT_MAJOR_VERSION) + { + snprintf(handle->m_lasterr, SCAP_LASTERR_SIZE, + "cannot correctly parse the capture. Upgrade your version."); + return SCAP_VERSION_MISMATCH; + } + + return SCAP_SUCCESS; +} + // // Parse the headers of a trace file and load the tables // @@ -2493,19 +2566,13 @@ int32_t scap_read_init(scap_t *handle, gzFile f) size_t readsize; size_t toread; int fseekres; - int8_t found_mi = 0; - int8_t found_pl = 0; - int8_t found_fdl = 0; - int8_t found_il = 0; - int8_t found_ul = 0; + int32_t rc; int8_t found_ev = 0; // // Read the section header block // - if(gzread(f, &bh, sizeof(bh)) != sizeof(bh) || - gzread(f, &sh, sizeof(sh)) != sizeof(sh) || - gzread(f, &bt, sizeof(bt)) != sizeof(bt)) + if(gzread(f, &bh, sizeof(bh)) != sizeof(bh)) { snprintf(handle->m_lasterr, SCAP_LASTERR_SIZE, "error reading from file (1)"); return SCAP_FAILURE; @@ -2517,17 +2584,9 @@ int32_t scap_read_init(scap_t *handle, gzFile f) return SCAP_FAILURE; } - if(sh.byte_order_magic != 0x1a2b3c4d) + if((rc = scap_read_section_header(handle, f)) != SCAP_SUCCESS) { - snprintf(handle->m_lasterr, SCAP_LASTERR_SIZE, "invalid magic number"); - return SCAP_FAILURE; - } - - if(sh.major_version > CURRENT_MAJOR_VERSION) - { - snprintf(handle->m_lasterr, SCAP_LASTERR_SIZE, - "cannot correctly parse the capture. Upgrade your version."); - return SCAP_VERSION_MISMATCH; + return rc; } // @@ -2541,8 +2600,7 @@ int32_t scap_read_init(scap_t *handle, gzFile f) // If we don't find the event block header, // it means there is no event in the file. // - if (readsize == 0 && !found_ev && found_mi && found_pl && - found_il && found_fdl && found_ul) + if (readsize == 0 && !found_ev) { snprintf(handle->m_lasterr, SCAP_LASTERR_SIZE, "no events in file"); return SCAP_FAILURE; @@ -2554,7 +2612,6 @@ int32_t scap_read_init(scap_t *handle, gzFile f) { case MI_BLOCK_TYPE: case MI_BLOCK_TYPE_INT: - found_mi = 1; if(scap_read_machine_info(handle, f, bh.block_total_length - sizeof(block_header) - 4) != SCAP_SUCCESS) { @@ -2573,7 +2630,6 @@ int32_t scap_read_init(scap_t *handle, gzFile f) case PL_BLOCK_TYPE_V1_INT: case PL_BLOCK_TYPE_V2_INT: case PL_BLOCK_TYPE_V3_INT: - found_pl = 1; if(scap_read_proclist(handle, f, bh.block_total_length - sizeof(block_header) - 4, bh.block_type) != SCAP_SUCCESS) { @@ -2583,7 +2639,6 @@ int32_t scap_read_init(scap_t *handle, gzFile f) case FDL_BLOCK_TYPE: case FDL_BLOCK_TYPE_INT: case FDL_BLOCK_TYPE_V2: - found_fdl = 1; if(scap_read_fdlist(handle, f, bh.block_total_length - sizeof(block_header) - 4, bh.block_type) != SCAP_SUCCESS) { @@ -2613,7 +2668,6 @@ int32_t scap_read_init(scap_t *handle, gzFile f) case IL_BLOCK_TYPE: case IL_BLOCK_TYPE_INT: case IL_BLOCK_TYPE_V2: - found_il = 1; if(scap_read_iflist(handle, f, bh.block_total_length - sizeof(block_header) - 4, bh.block_type) != SCAP_SUCCESS) { @@ -2623,7 +2677,6 @@ int32_t scap_read_init(scap_t *handle, gzFile f) case UL_BLOCK_TYPE: case UL_BLOCK_TYPE_INT: case UL_BLOCK_TYPE_V2: - found_ul = 1; if(scap_read_userlist(handle, f, bh.block_total_length - sizeof(block_header) - 4, bh.block_type) != SCAP_SUCCESS) { @@ -2666,23 +2719,11 @@ int32_t scap_read_init(scap_t *handle, gzFile f) } } - if(!found_mi) - { - snprintf(handle->m_lasterr, SCAP_LASTERR_SIZE, "corrupted input file. Can't find machine info block."); - return SCAP_FAILURE; - } - - if(!found_ul) - { - snprintf(handle->m_lasterr, SCAP_LASTERR_SIZE, "corrupted input file. Can't find user list block."); - return SCAP_FAILURE; - } - - if(!found_il) - { - snprintf(handle->m_lasterr, SCAP_LASTERR_SIZE, "corrupted input file. Can't find interface list block."); - return SCAP_FAILURE; - } + // + // NOTE: can't require a user list block, interface list block, or machine info block + // any longer--with the introduction of source plugins, it is legitimate to have + // trace files that don't contain those blocks + // return SCAP_SUCCESS; } @@ -2738,6 +2779,27 @@ int32_t scap_next_offline(scap_t *handle, OUT scap_evt **pevent, OUT uint16_t *p } } + // For the simple case of concatenating two capture + // files, where the two capture files were both + // generated by plugins, you might see section header + // blocks. Just skip them. + // + // If the concatenated capture file is from a + // non-plugin capture, this loop will run into one of + // the other block types (user list, process list, + // etc), and the read will still eventually fail. + if(bh.block_type == SHB_BLOCK_TYPE) + { + int32_t rc; + + if((rc = scap_read_section_header(handle, f)) != SCAP_SUCCESS) + { + return rc; + } + + return SCAP_TIMEOUT; + } + if(bh.block_type != EV_BLOCK_TYPE && bh.block_type != EV_BLOCK_TYPE_V2 && bh.block_type != EV_BLOCK_TYPE_INT && diff --git a/userspace/libscap/scap_savefile.h b/userspace/libscap/scap_savefile.h index 104d1f188..ebbcc16f3 100644 --- a/userspace/libscap/scap_savefile.h +++ b/userspace/libscap/scap_savefile.h @@ -72,7 +72,7 @@ typedef struct _section_header_block /////////////////////////////////////////////////////////////////////////////// #define MI_BLOCK_TYPE 0x201 #define MI_BLOCK_TYPE_INT 0x8002ABCD // This is the unofficial number used before the - // library release. We'll keep him for a while for + // library release. We'll keep it for a while for // backward compatibility /////////////////////////////////////////////////////////////////////////////// @@ -80,17 +80,17 @@ typedef struct _section_header_block /////////////////////////////////////////////////////////////////////////////// #define PL_BLOCK_TYPE_V1 0x202 #define PL_BLOCK_TYPE_V1_INT 0x8000ABCD // This is the unofficial number used before the - // library release. We'll keep him for a while for + // library release. We'll keep it for a while for // backward compatibility #define PL_BLOCK_TYPE_V2 0x207 #define PL_BLOCK_TYPE_V2_INT 0x8013ABCD // This is the unofficial number used before the - // library release. We'll keep him for a while for + // library release. We'll keep it for a while for // backward compatibility #define PL_BLOCK_TYPE_V3 0x209 #define PL_BLOCK_TYPE_V3_INT 0x8014ABCD // This is the unofficial number used before the - // library release. We'll keep him for a while for + // library release. We'll keep it for a while for // backward compatibility #define PL_BLOCK_TYPE_V4 0x210 @@ -110,7 +110,7 @@ typedef struct _section_header_block /////////////////////////////////////////////////////////////////////////////// #define FDL_BLOCK_TYPE 0x203 #define FDL_BLOCK_TYPE_INT 0x8001ABCD // This is the unofficial number used before the - // library release. We'll keep him for a while for + // library release. We'll keep it for a while for // backward compatibility #define FDL_BLOCK_TYPE_V2 0x218 @@ -119,7 +119,7 @@ typedef struct _section_header_block /////////////////////////////////////////////////////////////////////////////// #define EV_BLOCK_TYPE 0x204 #define EV_BLOCK_TYPE_INT 0x8010ABCD // This is the unofficial number used before the - // library release. We'll keep him for a while for + // library release. We'll keep it for a while for // backward compatibility #define EV_BLOCK_TYPE_V2 0x216 @@ -128,7 +128,7 @@ typedef struct _section_header_block /////////////////////////////////////////////////////////////////////////////// #define IL_BLOCK_TYPE 0x205 #define IL_BLOCK_TYPE_INT 0x8011ABCD // This is the unofficial number used before the - // library release. We'll keep him for a while for + // library release. We'll keep it for a while for // backward compatibility #define IL_BLOCK_TYPE_V2 0x219 @@ -137,7 +137,7 @@ typedef struct _section_header_block /////////////////////////////////////////////////////////////////////////////// #define UL_BLOCK_TYPE 0x206 #define UL_BLOCK_TYPE_INT 0x8012ABCD // This is the unofficial number used before the - // library release. We'll keep him for a while for + // library release. We'll keep it for a while for // backward compatibility #define UL_BLOCK_TYPE_V2 0x220 From 13ba8938f71a403d67771d7720a234295477cee4 Mon Sep 17 00:00:00 2001 From: Mark Stemm Date: Tue, 5 Oct 2021 14:02:32 -0700 Subject: [PATCH 029/148] libsinsp support for plugins libsinsp support for plugins involves: - methods to load a set of plugins and associate them with an inspector (add_plugin, get_plugins, get_plugin_by_id, get_source_plugin_by_source, etc.) - methods to open a stream of events from a plugin and check its progress (set_input_plugin/set_input_plugin_open_params, get_read_progress_plugin, etc.) - plugin_evt_processor is forward-looking and will allow for parallel reading of events from plugins. Co-authored-by: Loris Degioanni Co-authored-by: Leonardo Grasso Signed-off-by: Mark Stemm --- userspace/libsinsp/CMakeLists.txt | 1 + userspace/libsinsp/event.h | 2 + userspace/libsinsp/plugin_evt_processor.cpp | 284 ++++++++++++++++++++ userspace/libsinsp/plugin_evt_processor.h | 79 ++++++ userspace/libsinsp/settings.h | 6 + userspace/libsinsp/sinsp.cpp | 282 ++++++++++++++++--- userspace/libsinsp/sinsp.h | 62 ++++- 7 files changed, 671 insertions(+), 45 deletions(-) create mode 100755 userspace/libsinsp/plugin_evt_processor.cpp create mode 100755 userspace/libsinsp/plugin_evt_processor.h diff --git a/userspace/libsinsp/CMakeLists.txt b/userspace/libsinsp/CMakeLists.txt index cd7fc2820..f7d2efea8 100644 --- a/userspace/libsinsp/CMakeLists.txt +++ b/userspace/libsinsp/CMakeLists.txt @@ -79,6 +79,7 @@ set(SINSP_SOURCES logger.cpp parsers.cpp plugin.cpp + plugin_evt_processor.cpp prefix_search.cpp protodecoder.cpp threadinfo.cpp diff --git a/userspace/libsinsp/event.h b/userspace/libsinsp/event.h index ed2a6e2ea..4591e7669 100644 --- a/userspace/libsinsp/event.h +++ b/userspace/libsinsp/event.h @@ -527,6 +527,8 @@ VISIBILITY_PRIVATE friend class protocol_manager; friend class test_helpers::event_builder; friend class test_helpers::sinsp_mock; + friend class sinsp_pep_flt_worker; + friend class sinsp_plugin_evt_processor; }; /*@}*/ diff --git a/userspace/libsinsp/plugin_evt_processor.cpp b/userspace/libsinsp/plugin_evt_processor.cpp new file mode 100755 index 000000000..3f4ceed5c --- /dev/null +++ b/userspace/libsinsp/plugin_evt_processor.cpp @@ -0,0 +1,284 @@ +/* +Copyright (C) 2021 The Falco Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. + +*/ + +#include "sinsp.h" +#include "sinsp_int.h" +#include "filter.h" +#include "filterchecks.h" +#include "plugin.h" +#include "plugin_evt_processor.h" + +class sinsp_async_extractor_ctx; + +/////////////////////////////////////////////////////////////////////////////// +// sinsp_pep_flt_worker implementation +/////////////////////////////////////////////////////////////////////////////// +void worker_thread(sinsp_pep_flt_worker* This) +{ + while(This->m_die == false) + { + if(This->m_state == sinsp_pep_flt_worker::WS_WORKING) + { + This->process_event(); + This->m_state = sinsp_pep_flt_worker::WS_HAS_RESULT; + } + this_thread::yield(); + } +} + +sinsp_pep_flt_worker::sinsp_pep_flt_worker(sinsp* inspector, sinsp_plugin_evt_processor* pprocessor, bool async) +{ + m_inspector = inspector; + m_pprocessor = pprocessor; + m_evt.m_info = &(g_infotables.m_event_info[PPME_PLUGINEVENT_E]); + m_evt.m_inspector = m_inspector; + if(async) + { + m_th = new thread(worker_thread, this); + } + m_evt_storage.resize(128000); +} + +void sinsp_pep_flt_worker::set_filter(sinsp_filter* filter) +{ + m_filter = filter; +} + +sinsp_pep_flt_worker::~sinsp_pep_flt_worker() +{ + if(m_th) + { + m_die = true; + m_th->join(); + delete m_th; + } +} + +bool sinsp_pep_flt_worker::process_event() +{ + m_cnt++; + m_evt.m_filtered_out = false; + m_evt.m_flags = 0; + m_evt.m_poriginal_evt = NULL; + + m_evt.m_filtered_out = false; + if(m_filter && m_filter->run(&m_evt) == false) + { + m_evt.m_filtered_out = true; + } + +// m_evt.m_filtered_out = true; +// uint64_t r = 0; +// for(uint64_t j = 0; j < 10000; j++) +// { +// r += j; +// r *= (r + j); +// } +// m_tmp = r; + + return &m_evt; +} + +/////////////////////////////////////////////////////////////////////////////// +// sinsp_plugin_evt_processor implementation +/////////////////////////////////////////////////////////////////////////////// +sinsp_plugin_evt_processor::sinsp_plugin_evt_processor(sinsp* inspector) +{ + m_inspector = inspector; + init(); +} + +sinsp_plugin_evt_processor::~sinsp_plugin_evt_processor() +{ +} + +void sinsp_plugin_evt_processor::init() +{ + sinsp_filter* cf; + +#ifdef PARALLEL_PLUGIN_EVT_FILTERING_ENABLED + for(uint32_t j = 0; j < m_nworkers; j++) + { + m_workers.emplace_back(std::make_shared(m_inspector, this, true)); + } +#endif + + m_sync_worker = std::make_shared(m_inspector, this, false); +} + +void sinsp_plugin_evt_processor::compile(string filter) +{ + sinsp_filter* cf; + +#ifdef MULTITHREAD_PLUGIN_EVT_PROCESSOR_ENABLED + for(uint32_t j = 0; j < m_nworkers; j++) + { + m_inprogress = true; + m_inprogress_infos.clear(); + sinsp_filter_compiler wcompiler(m_inspector, filter); + cf = wcompiler.compile(); + m_inprogress = false; + + m_workers[j]->set_filter(cf); + } +#endif + + m_inprogress = true; + m_inprogress_infos.clear(); + sinsp_filter_compiler scompiler(m_inspector, filter); + cf = scompiler.compile(); + m_inprogress = false; + m_sync_worker->set_filter(cf); +} + +std::shared_ptr sinsp_plugin_evt_processor::get_plugin_source_info(uint32_t id) +{ + if(m_inprogress) + { + // + // Have we already allocated the required plugin? + // + auto it = m_inprogress_infos.find(id); + if(it == m_inprogress_infos.end()) + { + // + // Plugin not allocated yet, allocate and configure it now + // + + // + // Locate the sinsp_plugin object correspondng to this plugin ID + // + std::shared_ptr pplg = m_inspector->get_plugin_by_id(id); + if(!pplg) + { + // + // This should never happen + // + ASSERT(false); + throw sinsp_exception("cannot find plugin with ID " + to_string(id)); + } + + // XXX/mstemm can we use the existing plugin directly that has already been initialized? + // ss_plugin_info* newpsi = new ss_plugin_info; + // *newpsi = pplg->m_source_info; + + // // + // // Initialize the new plugin instance + // // + // newpsi->is_async_extractor_configured = false; + // newpsi->is_async_extractor_present = false; + + // if(newpsi->init != NULL) + // { + // int32_t init_res; + // newpsi->state = newpsi->init(NULL, &init_res); + // if(init_res != SCAP_SUCCESS) + // { + // throw sinsp_exception(string("unable to initialize plugin ") + newpsi->get_name()); + // } + // } + + m_inprogress_infos[id] = pplg; + return pplg; + } + else + { + // + // Plugin already allocated, point to the existing one + // + return it->second; + } + } + else + { + return std::shared_ptr(); + } +} + +#ifdef MULTITHREAD_PLUGIN_EVT_PROCESSOR_ENABLED +void sinsp_plugin_evt_processor::prepare_worker(sinsp_pep_flt_worker &w, sinsp_evt* evt) +{ + uint32_t pelen = evt->m_pevt->len; + + if(pelen > w.m_evt_storage.size()) + { + w.m_evt_storage.resize(pelen); + } + + memcpy(&(w.m_evt_storage[0]), evt->m_pevt, evt->m_pevt->len); + w.m_evt.m_pevt = (scap_evt*)&(w.m_evt_storage[0]); + w.m_evt.m_evtnum = evt->m_evtnum; +} +#else +void sinsp_plugin_evt_processor::prepare_worker(sinsp_pep_flt_worker &w, sinsp_evt* evt) +{ + uint32_t pelen = evt->m_pevt->len; + w.m_evt.m_pevt = evt->m_pevt; + w.m_evt.m_evtnum = evt->m_evtnum; +} +#endif + +sinsp_evt* sinsp_plugin_evt_processor::process_event(sinsp_evt* evt) +{ +#ifdef MULTITHREAD_PLUGIN_EVT_PROCESSOR_ENABLED + if(is_worker_available()) + { + for(auto w : m_workers) + { + if(w->m_state == sinsp_pep_flt_worker::WS_READY) + { + prepare_worker(*(w.get()), evt); + w->m_state = sinsp_pep_flt_worker::WS_WORKING; + return NULL; + } + } + } +#endif + + prepare_worker(*(m_sync_worker.get()), evt); + m_sync_worker->process_event(); + sinsp_evt* res = m_sync_worker->get_evt(); + return res; +} + +sinsp_evt* sinsp_plugin_evt_processor::get_event_from_backlog() +{ + for(auto w : m_workers) + { + if(w->m_state == sinsp_pep_flt_worker::WS_HAS_RESULT) + { + w->m_state = sinsp_pep_flt_worker::WS_READY; + sinsp_evt* res = w->get_evt(); + return res; + } + } + + return NULL; +} + +bool sinsp_plugin_evt_processor::is_worker_available() +{ + for(auto w : m_workers) + { + if(w->m_state == sinsp_pep_flt_worker::WS_READY) + { + return true; + } + } + + return false; +} diff --git a/userspace/libsinsp/plugin_evt_processor.h b/userspace/libsinsp/plugin_evt_processor.h new file mode 100755 index 000000000..60f3ad2f3 --- /dev/null +++ b/userspace/libsinsp/plugin_evt_processor.h @@ -0,0 +1,79 @@ +/* +Copyright (C) 2021 The Falco Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. + +*/ + +#pragma once + +#include +#include + +class sinsp_pep_flt_worker +{ +public: + enum pep_flt_worker_state + { + WS_READY, + WS_WORKING, + WS_HAS_RESULT, + }; + sinsp_pep_flt_worker(sinsp* inspector, sinsp_plugin_evt_processor* pprocessor, bool async); + void set_filter(sinsp_filter* filter); + ~sinsp_pep_flt_worker(); + inline bool process_event(); + + inline sinsp_evt* get_evt() + { + return &m_evt; + } + + sinsp* m_inspector; + sinsp_evt m_evt; + std::vector m_evt_storage; + sinsp_filter* m_filter = NULL; + thread* m_th = NULL; + bool m_die = false; + pep_flt_worker_state m_state = WS_READY; + sinsp_plugin_evt_processor* m_pprocessor; + uint32_t m_cnt = 0; + uint32_t m_tmp = 0; +}; + +class sinsp_plugin_evt_processor +{ +public: + sinsp_plugin_evt_processor(sinsp* inspector); + ~sinsp_plugin_evt_processor(); + void init(); + void compile(string filter); + sinsp_evt* process_event(sinsp_evt *evt); + sinsp_evt* get_event_from_backlog(); + std::shared_ptr get_plugin_source_info(uint32_t id); + +private: + void prepare_worker(sinsp_pep_flt_worker& w, sinsp_evt* evt); + bool is_worker_available(); + + sinsp* m_inspector; + uint32_t m_nworkers = 1; + std::vector> m_workers; + std::shared_ptr m_sync_worker; + std::vector> m_source_info_list; + bool m_inprogress = false; + map> m_inprogress_infos; + std::shared_ptr m_cur_source_info; + +friend class sinsp_pep_flt_worker; +}; diff --git a/userspace/libsinsp/settings.h b/userspace/libsinsp/settings.h index ba278fa7b..4242034c6 100644 --- a/userspace/libsinsp/settings.h +++ b/userspace/libsinsp/settings.h @@ -111,3 +111,9 @@ static const unsigned MAX_JSON_SIZE_B = 500 * 1024; // 500 kiB #define K8S_DATA_MAX_B 100 * 1024 * 1024 #define K8S_DATA_CHUNK_WAIT_US 1000 #define METADATA_DATA_WATCH_FREQ_SEC 1 + +// +// Define this to enable parallel (multi-threaded) filtering of events coming +// from plugins +// +#undef MULTITHREAD_PLUGIN_EVT_PROCESSOR_ENABLED diff --git a/userspace/libsinsp/sinsp.cpp b/userspace/libsinsp/sinsp.cpp index 0475eb326..9705d1485 100644 --- a/userspace/libsinsp/sinsp.cpp +++ b/userspace/libsinsp/sinsp.cpp @@ -35,6 +35,9 @@ limitations under the License. #include "cyclewriter.h" #include "protodecoder.h" #include "dns_manager.h" +#include "plugin.h" +#include "plugin_evt_processor.h" + #ifndef CYGWING_AGENT #ifndef MINIMAL_BUILD @@ -75,12 +78,14 @@ sinsp::sinsp(bool static_container, const std::string static_id, const std::stri #endif m_h = NULL; m_parser = NULL; + m_plugin_evt_processor = NULL; m_dumper = NULL; m_is_dumping = false; m_metaevt = NULL; m_meinfo.m_piscapevt = NULL; m_network_interfaces = NULL; m_parser = new sinsp_parser(this); + m_plugin_evt_processor = new sinsp_plugin_evt_processor(this); m_thread_manager = new sinsp_thread_manager(this); m_max_fdtable_size = MAX_FD_TABLE_SIZE; m_inactive_container_scan_time_ns = DEFAULT_INACTIVE_CONTAINER_SCAN_TIME_S * ONE_SECOND_IN_NS; @@ -162,6 +167,8 @@ sinsp::sinsp(bool static_container, const std::string static_id, const std::stri #endif // !defined(CYGWING_AGENT) && !defined(MINIMAL_BUILD) m_filter_proc_table_when_saving = false; + + m_n_async_plugin_extractors = 0; } sinsp::~sinsp() @@ -179,6 +186,12 @@ sinsp::~sinsp() m_parser = NULL; } + if(m_plugin_evt_processor) + { + delete m_plugin_evt_processor; + m_plugin_evt_processor = NULL; + } + if(m_thread_manager) { delete m_thread_manager; @@ -209,6 +222,7 @@ sinsp::~sinsp() sinsp_dns_manager::get().cleanup(); #endif #endif + m_plugins_list.clear(); } void sinsp::add_protodecoders() @@ -483,6 +497,19 @@ void sinsp::open_live_common(uint32_t timeout_ms, scap_mode_t mode) add_suppressed_comms(oargs); + // + // If a plugin was configured, pass it to scap and set the capture mode to + // SCAP_MODE_PLUGIN. + // + if(m_input_plugin) + { + sinsp_source_plugin *splugin = static_cast(m_input_plugin.get()); + oargs.input_plugin = splugin->plugin_info(); + oargs.input_plugin_params = (char*)m_input_plugin_open_params.c_str(); + m_mode = SCAP_MODE_PLUGIN; + oargs.mode = SCAP_MODE_PLUGIN; + } + int32_t scap_rc; m_h = scap_open(oargs, error, &scap_rc); @@ -1099,6 +1126,9 @@ int32_t sinsp::next(OUT sinsp_evt **puevt) { sinsp_evt* evt; int32_t res; +#ifdef MULTITHREAD_PLUGIN_EVT_PROCESSOR_ENABLED + bool from_plugin_proc_backlog = false; +#endif // // Check if there are fake cpu events to events @@ -1123,58 +1153,69 @@ int32_t sinsp::next(OUT sinsp_evt **puevt) #endif else { - evt = &m_evt; - - // - // Reset previous event's decoders if required - // - if(m_decoders_reset_list.size() != 0) +#ifdef MULTITHREAD_PLUGIN_EVT_PROCESSOR_ENABLED + evt = m_plugin_evt_processor->get_event_from_backlog(); + if(evt != NULL) { - vector::iterator it; - for(it = m_decoders_reset_list.begin(); it != m_decoders_reset_list.end(); ++it) - { - (*it)->on_reset(evt); - } - - m_decoders_reset_list.clear(); + res = SCAP_SUCCESS; + from_plugin_proc_backlog = true; } - - // - // Get the event from libscap - // - res = scap_next(m_h, &(evt->m_pevt), &(evt->m_cpuid)); - - if(res != SCAP_SUCCESS) + else +#endif { - if(res == SCAP_TIMEOUT) + evt = &m_evt; + + // + // Reset previous event's decoders if required + // + if(m_decoders_reset_list.size() != 0) { - if (m_external_event_processor) + vector::iterator it; + for(it = m_decoders_reset_list.begin(); it != m_decoders_reset_list.end(); ++it) { - m_external_event_processor->process_event(NULL, libsinsp::EVENT_RETURN_TIMEOUT); + (*it)->on_reset(evt); } - *puevt = NULL; - return res; + + m_decoders_reset_list.clear(); } - else if(res == SCAP_EOF) + + // + // Get the event from libscap + // + res = scap_next(m_h, &(evt->m_pevt), &(evt->m_cpuid)); + + if(res != SCAP_SUCCESS) { - if (m_external_event_processor) + if(res == SCAP_TIMEOUT) { - m_external_event_processor->process_event(NULL, libsinsp::EVENT_RETURN_EOF); + if (m_external_event_processor) + { + m_external_event_processor->process_event(NULL, libsinsp::EVENT_RETURN_TIMEOUT); + } + *puevt = NULL; + return res; } - } - else if(res == SCAP_UNEXPECTED_BLOCK) - { - uint64_t filepos = scap_ftell(m_h) - scap_get_unexpected_block_readsize(m_h); - restart_capture_at_filepos(filepos); - return SCAP_TIMEOUT; + else if(res == SCAP_EOF) + { + if (m_external_event_processor) + { + m_external_event_processor->process_event(NULL, libsinsp::EVENT_RETURN_EOF); + } + } + else if(res == SCAP_UNEXPECTED_BLOCK) + { + uint64_t filepos = scap_ftell(m_h) - scap_get_unexpected_block_readsize(m_h); + restart_capture_at_filepos(filepos); + return SCAP_TIMEOUT; - } - else - { - m_lasterr = scap_getlasterr(m_h); - } + } + else + { + m_lasterr = scap_getlasterr(m_h); + } - return res; + return res; + } } } @@ -1314,7 +1355,24 @@ int32_t sinsp::next(OUT sinsp_evt **puevt) return SCAP_TIMEOUT; } #else - m_parser->process_event(evt); + if(evt->get_type() == PPME_PLUGINEVENT_E) + { +#ifdef MULTITHREAD_PLUGIN_EVT_PROCESSOR_ENABLED + if(!from_plugin_proc_backlog) +#endif // MULTITHREAD_PLUGIN_EVT_PROCESSOR_ENABLED + { + evt = m_plugin_evt_processor->process_event(evt); + if(evt == NULL) + { + *puevt = evt; + return SCAP_TIMEOUT; + } + } + } + else + { + m_parser->process_event(evt); + } #endif // @@ -1585,6 +1643,90 @@ void sinsp::set_statsd_port(const uint16_t port) } } +void sinsp::add_plugin(std::shared_ptr plugin) +{ + uint32_t ncpus = thread::hardware_concurrency(); + bool avoid_async = ncpus == 0 || (m_n_async_plugin_extractors >= (ncpus - 1)); + + if(avoid_async) + { + plugin->disable_async_extract(); + } + + for(auto& it : m_plugins_list) + { + if(it->name() == plugin->name()) + { + throw sinsp_exception("found multiple plugins with name " + it->name() + ". Aborting."); + } + } + + m_plugins_list.push_back(plugin); +} + +void sinsp::set_input_plugin(string plugin_name) +{ + for(auto& it : m_plugins_list) + { + if(it->name() == plugin_name) + { + if(it->type() != TYPE_SOURCE_PLUGIN) + { + throw sinsp_exception("plugin " + plugin_name + " is not a source plugin and cannot be used as input."); + } + + m_input_plugin = it; + return; + } + } + + throw sinsp_exception("plugin " + plugin_name + " does not exist"); +} + +void sinsp::set_input_plugin_open_params(string params) +{ + m_input_plugin_open_params = params; +} + +const std::vector>& sinsp::get_plugins() +{ + return m_plugins_list; +} + +std::shared_ptr sinsp::get_plugin_by_id(uint32_t plugin_id) +{ + for(auto &it : m_plugins_list) + { + if(it->type() == TYPE_SOURCE_PLUGIN) + { + sinsp_source_plugin *splugin = static_cast(it.get()); + if(splugin->id() == plugin_id) + { + return it; + } + } + } + + return std::shared_ptr(); +} + +std::shared_ptr sinsp::get_source_plugin_by_source(const std::string &source) +{ + for(auto &it : m_plugins_list) + { + if(it->type() == TYPE_SOURCE_PLUGIN) + { + sinsp_source_plugin *splugin = static_cast(it.get()); + if(splugin->event_source() == source) + { + return it; + } + } + } + + return std::shared_ptr(); +} + void sinsp::stop_capture() { if(scap_stop_capture(m_h) != SCAP_SUCCESS) @@ -1652,6 +1794,8 @@ void sinsp::set_filter(const string& filter) sinsp_filter_compiler compiler(this, filter); m_filter = compiler.compile(); m_filterstring = filter; + + m_plugin_evt_processor->compile(filter); } const string sinsp::get_filter() @@ -1921,7 +2065,7 @@ bool sinsp::setup_cycle_writer(string base_file_name, int rollover_mb, int durat return m_cycle_writer->setup(base_file_name, rollover_mb, duration_seconds, file_limit, event_limit, &m_dumper); } -double sinsp::get_read_progress() +double sinsp::get_read_progress_file() { if(m_input_fd != 0) { @@ -1956,6 +2100,60 @@ void sinsp::set_metadata_download_params(uint32_t data_max_b, m_metadata_download_params.m_data_watch_freq_sec = data_watch_freq_sec; } +void sinsp::get_read_progress_plugin(OUT double* nres, string* sres) +{ + ASSERT(nres != NULL); + ASSERT(sres != NULL); + if(!nres || !sres) + { + return; + } + + if (!m_input_plugin) + { + *nres = -1; + *sres = "No Input Plugin"; + + return; + } + + sinsp_source_plugin *splugin = static_cast(m_input_plugin.get()); + + uint32_t nplg; + *sres = splugin->get_progress(nplg); + + *nres = ((double)nplg) / 100; +} + +double sinsp::get_read_progress() +{ + if(is_plugin()) + { + double res = 0; + get_read_progress_plugin(&res, NULL); + return res; + } + else + { + return get_read_progress_file(); + } +} + +double sinsp::get_read_progress_with_str(OUT string* progress_str) +{ + if(is_plugin()) + { + double res; + get_read_progress_plugin(&res, progress_str); + return res; + } + else + { + *progress_str = ""; + return get_read_progress_file(); + } +} + bool sinsp::remove_inactive_threads() { return m_thread_manager->remove_inactive_threads(); diff --git a/userspace/libsinsp/sinsp.h b/userspace/libsinsp/sinsp.h index 595c2ec5e..7f2a24f72 100644 --- a/userspace/libsinsp/sinsp.h +++ b/userspace/libsinsp/sinsp.h @@ -89,6 +89,10 @@ using namespace std; #define ONE_SECOND_IN_NS 1000000000LL +#ifdef _WIN32 +#define NOCURSESUI +#endif + #include "tuples.h" #include "fdinfo.h" #include "threadinfo.h" @@ -108,6 +112,8 @@ class k8s; #endif // !defined(CYGWING_AGENT) && !defined(MINIMAL_BUILD) class sinsp_partial_tracer; class mesos; +class sinsp_plugin; +class sinsp_plugin_evt_processor; #if defined(HAS_CAPTURE) && !defined(_WIN32) class sinsp_ssl; @@ -619,6 +625,14 @@ class SINSP_PUBLIC sinsp : public capture_stats_source, public wmi_handle_source return m_mode == SCAP_MODE_NODRIVER; } + /*! + \brief Returns true if the current capture has a plugin producing events + */ + inline bool is_plugin() + { + return m_mode == SCAP_MODE_PLUGIN; + } + /*! \brief Returns true if truncated environments should be loaded from /proc */ @@ -769,11 +783,18 @@ class SINSP_PUBLIC sinsp : public capture_stats_source, public wmi_handle_source void unset_eventmask(uint32_t event_id); /*! - \brief When reading events from a trace file, this function returns the - read progress as a number between 0 and 100. + \brief When reading events from a trace file or a plugin, this function + returns the read progress as a number between 0 and 100. */ double get_read_progress(); + /*! + \brief When reading events from a trace file or a plugin, this function + returns the read progress as a number and as a string, giving the plugins + flexibility on the format. + */ + double get_read_progress_with_str(OUT string* progress_str); + /*! \brief Make the amount of data gathered for a syscall to be determined by the number of parameters. @@ -916,6 +937,17 @@ class SINSP_PUBLIC sinsp : public capture_stats_source, public wmi_handle_source void set_cri_delay(uint64_t delay_ms); void set_container_labels_max_len(uint32_t max_label_len); + void add_plugin(std::shared_ptr plugin); + void set_input_plugin(string plugin_name); + void set_input_plugin_open_params(string params); + const std::vector>& get_plugins(); + std::shared_ptr get_plugin_by_id(uint32_t plugin_id); + std::shared_ptr get_source_plugin_by_source(const std::string &source); + sinsp_plugin_evt_processor* get_plugin_evt_processor() + { + return m_plugin_evt_processor; + } + uint64_t get_lastevent_ts() const { return m_lastevent_ts; } VISIBILITY_PROTECTED @@ -987,10 +1019,13 @@ VISIBILITY_PRIVATE m_increased_snaplen_port_range.range_end > 0; } + double get_read_progress_file(); + void get_read_progress_plugin(OUT double* nres, string* sres); + void get_procs_cpu_from_driver(uint64_t ts); scap_t* m_h; - uint32_t m_nevts; + uint64_t m_nevts; int64_t m_filesize; scap_mode_t m_mode = SCAP_MODE_NONE; @@ -1018,6 +1053,7 @@ VISIBILITY_PRIVATE uint64_t m_lastevent_ts; // the parsing engine sinsp_parser* m_parser; + sinsp_plugin_evt_processor* m_plugin_evt_processor; // the statistics analysis engine scap_dumper_t* m_dumper; bool m_is_dumping; @@ -1201,6 +1237,26 @@ VISIBILITY_PRIVATE // returned in sinsp::next() std::set m_suppressed_comms; + // + // List of the sinsp/scap plugins configured by the user. + // + std::vector> m_plugins_list; + // + // Count of plugins that are using a full CPU to accelerate field + // extraction. We want this to be lower than the number of available CPUs. + // + uint32_t m_n_async_plugin_extractors; + // + // The ID of the plugin to use as event input, or zero + // if no source plugin should be used as source + // + std::shared_ptr m_input_plugin; + // + // String with the parameters for the plugin to be used as input. + // These parameters will be passed to the open function of the plugin. + // + string m_input_plugin_open_params; + friend class sinsp_parser; friend class sinsp_analyzer; friend class sinsp_analyzer_parsers; From 54ab5d2304637c51b68286460e4c6f6e639718a7 Mon Sep 17 00:00:00 2001 From: Mark Stemm Date: Tue, 5 Oct 2021 14:08:09 -0700 Subject: [PATCH 030/148] New filtercheck field evt.datetime.s This is like evt.datetime but the the time part skips nanoseconds. This can be used for events from plugins, which might not need nanosecond resolution. Co-authored-by: Loris Degioanni Co-authored-by: Leonardo Grasso Signed-off-by: Mark Stemm --- userspace/libsinsp/filterchecks.cpp | 21 ++++- userspace/libsinsp/filterchecks.h | 129 ++++++++++++++-------------- 2 files changed, 86 insertions(+), 64 deletions(-) diff --git a/userspace/libsinsp/filterchecks.cpp b/userspace/libsinsp/filterchecks.cpp index 132c6d5a1..89c0248d7 100644 --- a/userspace/libsinsp/filterchecks.cpp +++ b/userspace/libsinsp/filterchecks.cpp @@ -2793,6 +2793,7 @@ const filtercheck_field_info sinsp_filter_check_event_fields[] = {PT_CHARBUF, EPF_NONE, PF_NA, "evt.time.s", "event timestamp as a time string with no nanoseconds."}, {PT_CHARBUF, EPF_NONE, PF_NA, "evt.time.iso8601", "event timestamp in ISO 8601 format, including nanoseconds and time zone offset (in UTC)."}, {PT_CHARBUF, EPF_NONE, PF_NA, "evt.datetime", "event timestamp as a time string that includes the date."}, + {PT_CHARBUF, EPF_NONE, PF_NA, "evt.datetime.s", "event timestamp as a datetime string with no nanoseconds."}, {PT_ABSTIME, EPF_NONE, PF_DEC, "evt.rawtime", "absolute event timestamp, i.e. nanoseconds from epoch."}, {PT_ABSTIME, EPF_NONE, PF_DEC, "evt.rawtime.s", "integer part of the event timestamp (e.g. seconds since epoch)."}, {PT_ABSTIME, EPF_NONE, PF_10_PADDED_DEC, "evt.rawtime.ns", "fractional part of the absolute event timestamp."}, @@ -3343,6 +3344,7 @@ Json::Value sinsp_filter_check_event::extract_as_js(sinsp_evt *evt, OUT uint32_t case TYPE_TIME_S: case TYPE_TIME_ISO8601: case TYPE_DATETIME: + case TYPE_DATETIME_S: case TYPE_RUNTIME_TIME_OUTPUT_FORMAT: return (Json::Value::Int64)evt->get_ts(); @@ -3435,6 +3437,9 @@ uint8_t* sinsp_filter_check_event::extract(sinsp_evt *evt, OUT uint32_t* len, bo case TYPE_DATETIME: sinsp_utils::ts_to_string(evt->get_ts(), &m_strstorage, true, true); RETURN_EXTRACT_STRING(m_strstorage); + case TYPE_DATETIME_S: + sinsp_utils::ts_to_string(evt->get_ts(), &m_strstorage, true, false); + RETURN_EXTRACT_STRING(m_strstorage); case TYPE_RAWTS: RETURN_EXTRACT_VAR(evt->m_pevt->ts); case TYPE_RAWTS_S: @@ -6660,7 +6665,21 @@ char* sinsp_filter_check_reference::format_bytes(double val, uint32_t str_len, b char* sinsp_filter_check_reference::format_time(uint64_t val, uint32_t str_len) { - if(val >= ONE_SECOND_IN_NS) + if(val >= 3600 * ONE_SECOND_IN_NS) + { + snprintf(m_getpropertystr_storage, + sizeof(m_getpropertystr_storage), + "%.2u:%.2u:%.2u", (unsigned int)(val / (3600 * ONE_SECOND_IN_NS)), + (unsigned int)((val / (60 * ONE_SECOND_IN_NS)) % 60 ), + (unsigned int)((val / ONE_SECOND_IN_NS) % 60)); + } + else if(val >= 60 * ONE_SECOND_IN_NS) + { + snprintf(m_getpropertystr_storage, + sizeof(m_getpropertystr_storage), + "%u:%u", (unsigned int)(val / (60 * ONE_SECOND_IN_NS)), (unsigned int)((val / ONE_SECOND_IN_NS) % 60)); + } + else if(val >= ONE_SECOND_IN_NS) { snprintf(m_getpropertystr_storage, sizeof(m_getpropertystr_storage), diff --git a/userspace/libsinsp/filterchecks.h b/userspace/libsinsp/filterchecks.h index 0e67bcfc0..4e886504f 100644 --- a/userspace/libsinsp/filterchecks.h +++ b/userspace/libsinsp/filterchecks.h @@ -411,69 +411,72 @@ class sinsp_filter_check_event : public sinsp_filter_check TYPE_TIME_S = 2, TYPE_TIME_ISO8601 = 3, TYPE_DATETIME = 4, - TYPE_RAWTS = 5, - TYPE_RAWTS_S = 6, - TYPE_RAWTS_NS = 7, - TYPE_RELTS = 8, - TYPE_RELTS_S = 9, - TYPE_RELTS_NS = 10, - TYPE_LATENCY = 11, - TYPE_LATENCY_S = 12, - TYPE_LATENCY_NS = 13, - TYPE_LATENCY_QUANTIZED = 14, - TYPE_LATENCY_HUMAN = 15, - TYPE_DELTA = 16, - TYPE_DELTA_S = 17, - TYPE_DELTA_NS = 18, - TYPE_RUNTIME_TIME_OUTPUT_FORMAT = 19, - TYPE_DIR = 20, - TYPE_TYPE = 21, - TYPE_TYPE_IS = 22, - TYPE_SYSCALL_TYPE = 23, - TYPE_CATEGORY = 24, - TYPE_CPU = 25, - TYPE_ARGS = 26, - TYPE_ARGSTR = 27, - TYPE_ARGRAW = 28, - TYPE_INFO = 29, - TYPE_BUFFER = 30, - TYPE_BUFLEN = 31, - TYPE_RESSTR = 32, - TYPE_RESRAW = 33, - TYPE_FAILED = 34, - TYPE_ISIO = 35, - TYPE_ISIO_READ = 36, - TYPE_ISIO_WRITE = 37, - TYPE_IODIR = 38, - TYPE_ISWAIT = 39, - TYPE_WAIT_LATENCY = 40, - TYPE_ISSYSLOG = 41, - TYPE_COUNT = 42, - TYPE_COUNT_ERROR = 43, - TYPE_COUNT_ERROR_FILE = 44, - TYPE_COUNT_ERROR_NET = 45, - TYPE_COUNT_ERROR_MEMORY = 46, - TYPE_COUNT_ERROR_OTHER = 47, - TYPE_COUNT_EXIT = 48, - TYPE_COUNT_PROCINFO = 49, - TYPE_COUNT_THREADINFO = 50, - TYPE_AROUND = 51, - TYPE_ABSPATH = 52, - TYPE_BUFLEN_IN = 53, - TYPE_BUFLEN_OUT = 54, - TYPE_BUFLEN_FILE = 55, - TYPE_BUFLEN_FILE_IN = 56, - TYPE_BUFLEN_FILE_OUT = 57, - TYPE_BUFLEN_NET = 58, - TYPE_BUFLEN_NET_IN = 59, - TYPE_BUFLEN_NET_OUT = 60, - TYPE_ISOPEN_READ = 61, - TYPE_ISOPEN_WRITE = 62, - TYPE_INFRA_DOCKER_NAME = 63, - TYPE_INFRA_DOCKER_CONTAINER_ID = 64, - TYPE_INFRA_DOCKER_CONTAINER_NAME = 65, - TYPE_INFRA_DOCKER_CONTAINER_IMAGE = 66, - TYPE_ISOPEN_EXEC = 67, + TYPE_DATETIME_S = 5, + TYPE_RAWTS = 6, + TYPE_RAWTS_S = 7, + TYPE_RAWTS_NS = 8, + TYPE_RELTS = 9, + TYPE_RELTS_S = 10, + TYPE_RELTS_NS = 11, + TYPE_LATENCY = 12, + TYPE_LATENCY_S = 13, + TYPE_LATENCY_NS = 14, + TYPE_LATENCY_QUANTIZED = 15, + TYPE_LATENCY_HUMAN = 16, + TYPE_DELTA = 17, + TYPE_DELTA_S = 18, + TYPE_DELTA_NS = 19, + TYPE_RUNTIME_TIME_OUTPUT_FORMAT = 20, + TYPE_DIR = 21, + TYPE_TYPE = 22, + TYPE_TYPE_IS = 23, + TYPE_SYSCALL_TYPE = 24, + TYPE_CATEGORY = 25, + TYPE_CPU = 26, + TYPE_ARGS = 27, + TYPE_ARGSTR = 28, + TYPE_ARGRAW = 29, + TYPE_INFO = 30, + TYPE_BUFFER = 31, + TYPE_BUFLEN = 32, + TYPE_RESSTR = 33, + TYPE_RESRAW = 34, + TYPE_FAILED = 35, + TYPE_ISIO = 36, + TYPE_ISIO_READ = 37, + TYPE_ISIO_WRITE = 38, + TYPE_IODIR = 39, + TYPE_ISWAIT = 40, + TYPE_WAIT_LATENCY = 41, + TYPE_ISSYSLOG = 42, + TYPE_COUNT = 43, + TYPE_COUNT_ERROR = 44, + TYPE_COUNT_ERROR_FILE = 45, + TYPE_COUNT_ERROR_NET = 46, + TYPE_COUNT_ERROR_MEMORY = 47, + TYPE_COUNT_ERROR_OTHER = 48, + TYPE_COUNT_EXIT = 49, + TYPE_COUNT_PROCINFO = 50, + TYPE_COUNT_THREADINFO = 51, + TYPE_AROUND = 52, + TYPE_ABSPATH = 53, + TYPE_BUFLEN_IN = 54, + TYPE_BUFLEN_OUT = 55, + TYPE_BUFLEN_FILE = 56, + TYPE_BUFLEN_FILE_IN = 57, + TYPE_BUFLEN_FILE_OUT = 58, + TYPE_BUFLEN_NET = 59, + TYPE_BUFLEN_NET_IN = 60, + TYPE_BUFLEN_NET_OUT = 61, + TYPE_ISOPEN_READ = 62, + TYPE_ISOPEN_WRITE = 63, + TYPE_INFRA_DOCKER_NAME = 64, + TYPE_INFRA_DOCKER_CONTAINER_ID = 65, + TYPE_INFRA_DOCKER_CONTAINER_NAME = 66, + TYPE_INFRA_DOCKER_CONTAINER_IMAGE = 67, + TYPE_ISOPEN_EXEC = 68, + TYPE_PLUGIN_NAME = 69, + TYPE_PLUGIN_INFO = 70, }; sinsp_filter_check_event(); From 8ac46c87ef8ff00531710ab79123cfad57fabcf2 Mon Sep 17 00:00:00 2001 From: Mark Stemm Date: Fri, 8 Oct 2021 10:24:19 -0700 Subject: [PATCH 031/148] Remove plugin_evt_processor for now Was future-thinking towards parallel processing of events but we don't need it for MVP. Signed-off-by: Mark Stemm --- userspace/libsinsp/CMakeLists.txt | 1 - userspace/libsinsp/event.h | 2 - userspace/libsinsp/plugin.cpp | 1 - userspace/libsinsp/plugin_evt_processor.cpp | 284 -------------------- userspace/libsinsp/plugin_evt_processor.h | 79 ------ userspace/libsinsp/settings.h | 6 - userspace/libsinsp/sinsp.cpp | 124 +++------ userspace/libsinsp/sinsp.h | 6 - 8 files changed, 41 insertions(+), 462 deletions(-) delete mode 100755 userspace/libsinsp/plugin_evt_processor.cpp delete mode 100755 userspace/libsinsp/plugin_evt_processor.h diff --git a/userspace/libsinsp/CMakeLists.txt b/userspace/libsinsp/CMakeLists.txt index f7d2efea8..cd7fc2820 100644 --- a/userspace/libsinsp/CMakeLists.txt +++ b/userspace/libsinsp/CMakeLists.txt @@ -79,7 +79,6 @@ set(SINSP_SOURCES logger.cpp parsers.cpp plugin.cpp - plugin_evt_processor.cpp prefix_search.cpp protodecoder.cpp threadinfo.cpp diff --git a/userspace/libsinsp/event.h b/userspace/libsinsp/event.h index 4591e7669..ed2a6e2ea 100644 --- a/userspace/libsinsp/event.h +++ b/userspace/libsinsp/event.h @@ -527,8 +527,6 @@ VISIBILITY_PRIVATE friend class protocol_manager; friend class test_helpers::event_builder; friend class test_helpers::sinsp_mock; - friend class sinsp_pep_flt_worker; - friend class sinsp_plugin_evt_processor; }; /*@}*/ diff --git a/userspace/libsinsp/plugin.cpp b/userspace/libsinsp/plugin.cpp index e10658f26..707544394 100755 --- a/userspace/libsinsp/plugin.cpp +++ b/userspace/libsinsp/plugin.cpp @@ -29,7 +29,6 @@ limitations under the License. #include "filter.h" #include "filterchecks.h" #include "plugin.h" -#include "plugin_evt_processor.h" #include diff --git a/userspace/libsinsp/plugin_evt_processor.cpp b/userspace/libsinsp/plugin_evt_processor.cpp deleted file mode 100755 index 3f4ceed5c..000000000 --- a/userspace/libsinsp/plugin_evt_processor.cpp +++ /dev/null @@ -1,284 +0,0 @@ -/* -Copyright (C) 2021 The Falco Authors. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. - -*/ - -#include "sinsp.h" -#include "sinsp_int.h" -#include "filter.h" -#include "filterchecks.h" -#include "plugin.h" -#include "plugin_evt_processor.h" - -class sinsp_async_extractor_ctx; - -/////////////////////////////////////////////////////////////////////////////// -// sinsp_pep_flt_worker implementation -/////////////////////////////////////////////////////////////////////////////// -void worker_thread(sinsp_pep_flt_worker* This) -{ - while(This->m_die == false) - { - if(This->m_state == sinsp_pep_flt_worker::WS_WORKING) - { - This->process_event(); - This->m_state = sinsp_pep_flt_worker::WS_HAS_RESULT; - } - this_thread::yield(); - } -} - -sinsp_pep_flt_worker::sinsp_pep_flt_worker(sinsp* inspector, sinsp_plugin_evt_processor* pprocessor, bool async) -{ - m_inspector = inspector; - m_pprocessor = pprocessor; - m_evt.m_info = &(g_infotables.m_event_info[PPME_PLUGINEVENT_E]); - m_evt.m_inspector = m_inspector; - if(async) - { - m_th = new thread(worker_thread, this); - } - m_evt_storage.resize(128000); -} - -void sinsp_pep_flt_worker::set_filter(sinsp_filter* filter) -{ - m_filter = filter; -} - -sinsp_pep_flt_worker::~sinsp_pep_flt_worker() -{ - if(m_th) - { - m_die = true; - m_th->join(); - delete m_th; - } -} - -bool sinsp_pep_flt_worker::process_event() -{ - m_cnt++; - m_evt.m_filtered_out = false; - m_evt.m_flags = 0; - m_evt.m_poriginal_evt = NULL; - - m_evt.m_filtered_out = false; - if(m_filter && m_filter->run(&m_evt) == false) - { - m_evt.m_filtered_out = true; - } - -// m_evt.m_filtered_out = true; -// uint64_t r = 0; -// for(uint64_t j = 0; j < 10000; j++) -// { -// r += j; -// r *= (r + j); -// } -// m_tmp = r; - - return &m_evt; -} - -/////////////////////////////////////////////////////////////////////////////// -// sinsp_plugin_evt_processor implementation -/////////////////////////////////////////////////////////////////////////////// -sinsp_plugin_evt_processor::sinsp_plugin_evt_processor(sinsp* inspector) -{ - m_inspector = inspector; - init(); -} - -sinsp_plugin_evt_processor::~sinsp_plugin_evt_processor() -{ -} - -void sinsp_plugin_evt_processor::init() -{ - sinsp_filter* cf; - -#ifdef PARALLEL_PLUGIN_EVT_FILTERING_ENABLED - for(uint32_t j = 0; j < m_nworkers; j++) - { - m_workers.emplace_back(std::make_shared(m_inspector, this, true)); - } -#endif - - m_sync_worker = std::make_shared(m_inspector, this, false); -} - -void sinsp_plugin_evt_processor::compile(string filter) -{ - sinsp_filter* cf; - -#ifdef MULTITHREAD_PLUGIN_EVT_PROCESSOR_ENABLED - for(uint32_t j = 0; j < m_nworkers; j++) - { - m_inprogress = true; - m_inprogress_infos.clear(); - sinsp_filter_compiler wcompiler(m_inspector, filter); - cf = wcompiler.compile(); - m_inprogress = false; - - m_workers[j]->set_filter(cf); - } -#endif - - m_inprogress = true; - m_inprogress_infos.clear(); - sinsp_filter_compiler scompiler(m_inspector, filter); - cf = scompiler.compile(); - m_inprogress = false; - m_sync_worker->set_filter(cf); -} - -std::shared_ptr sinsp_plugin_evt_processor::get_plugin_source_info(uint32_t id) -{ - if(m_inprogress) - { - // - // Have we already allocated the required plugin? - // - auto it = m_inprogress_infos.find(id); - if(it == m_inprogress_infos.end()) - { - // - // Plugin not allocated yet, allocate and configure it now - // - - // - // Locate the sinsp_plugin object correspondng to this plugin ID - // - std::shared_ptr pplg = m_inspector->get_plugin_by_id(id); - if(!pplg) - { - // - // This should never happen - // - ASSERT(false); - throw sinsp_exception("cannot find plugin with ID " + to_string(id)); - } - - // XXX/mstemm can we use the existing plugin directly that has already been initialized? - // ss_plugin_info* newpsi = new ss_plugin_info; - // *newpsi = pplg->m_source_info; - - // // - // // Initialize the new plugin instance - // // - // newpsi->is_async_extractor_configured = false; - // newpsi->is_async_extractor_present = false; - - // if(newpsi->init != NULL) - // { - // int32_t init_res; - // newpsi->state = newpsi->init(NULL, &init_res); - // if(init_res != SCAP_SUCCESS) - // { - // throw sinsp_exception(string("unable to initialize plugin ") + newpsi->get_name()); - // } - // } - - m_inprogress_infos[id] = pplg; - return pplg; - } - else - { - // - // Plugin already allocated, point to the existing one - // - return it->second; - } - } - else - { - return std::shared_ptr(); - } -} - -#ifdef MULTITHREAD_PLUGIN_EVT_PROCESSOR_ENABLED -void sinsp_plugin_evt_processor::prepare_worker(sinsp_pep_flt_worker &w, sinsp_evt* evt) -{ - uint32_t pelen = evt->m_pevt->len; - - if(pelen > w.m_evt_storage.size()) - { - w.m_evt_storage.resize(pelen); - } - - memcpy(&(w.m_evt_storage[0]), evt->m_pevt, evt->m_pevt->len); - w.m_evt.m_pevt = (scap_evt*)&(w.m_evt_storage[0]); - w.m_evt.m_evtnum = evt->m_evtnum; -} -#else -void sinsp_plugin_evt_processor::prepare_worker(sinsp_pep_flt_worker &w, sinsp_evt* evt) -{ - uint32_t pelen = evt->m_pevt->len; - w.m_evt.m_pevt = evt->m_pevt; - w.m_evt.m_evtnum = evt->m_evtnum; -} -#endif - -sinsp_evt* sinsp_plugin_evt_processor::process_event(sinsp_evt* evt) -{ -#ifdef MULTITHREAD_PLUGIN_EVT_PROCESSOR_ENABLED - if(is_worker_available()) - { - for(auto w : m_workers) - { - if(w->m_state == sinsp_pep_flt_worker::WS_READY) - { - prepare_worker(*(w.get()), evt); - w->m_state = sinsp_pep_flt_worker::WS_WORKING; - return NULL; - } - } - } -#endif - - prepare_worker(*(m_sync_worker.get()), evt); - m_sync_worker->process_event(); - sinsp_evt* res = m_sync_worker->get_evt(); - return res; -} - -sinsp_evt* sinsp_plugin_evt_processor::get_event_from_backlog() -{ - for(auto w : m_workers) - { - if(w->m_state == sinsp_pep_flt_worker::WS_HAS_RESULT) - { - w->m_state = sinsp_pep_flt_worker::WS_READY; - sinsp_evt* res = w->get_evt(); - return res; - } - } - - return NULL; -} - -bool sinsp_plugin_evt_processor::is_worker_available() -{ - for(auto w : m_workers) - { - if(w->m_state == sinsp_pep_flt_worker::WS_READY) - { - return true; - } - } - - return false; -} diff --git a/userspace/libsinsp/plugin_evt_processor.h b/userspace/libsinsp/plugin_evt_processor.h deleted file mode 100755 index 60f3ad2f3..000000000 --- a/userspace/libsinsp/plugin_evt_processor.h +++ /dev/null @@ -1,79 +0,0 @@ -/* -Copyright (C) 2021 The Falco Authors. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. - -*/ - -#pragma once - -#include -#include - -class sinsp_pep_flt_worker -{ -public: - enum pep_flt_worker_state - { - WS_READY, - WS_WORKING, - WS_HAS_RESULT, - }; - sinsp_pep_flt_worker(sinsp* inspector, sinsp_plugin_evt_processor* pprocessor, bool async); - void set_filter(sinsp_filter* filter); - ~sinsp_pep_flt_worker(); - inline bool process_event(); - - inline sinsp_evt* get_evt() - { - return &m_evt; - } - - sinsp* m_inspector; - sinsp_evt m_evt; - std::vector m_evt_storage; - sinsp_filter* m_filter = NULL; - thread* m_th = NULL; - bool m_die = false; - pep_flt_worker_state m_state = WS_READY; - sinsp_plugin_evt_processor* m_pprocessor; - uint32_t m_cnt = 0; - uint32_t m_tmp = 0; -}; - -class sinsp_plugin_evt_processor -{ -public: - sinsp_plugin_evt_processor(sinsp* inspector); - ~sinsp_plugin_evt_processor(); - void init(); - void compile(string filter); - sinsp_evt* process_event(sinsp_evt *evt); - sinsp_evt* get_event_from_backlog(); - std::shared_ptr get_plugin_source_info(uint32_t id); - -private: - void prepare_worker(sinsp_pep_flt_worker& w, sinsp_evt* evt); - bool is_worker_available(); - - sinsp* m_inspector; - uint32_t m_nworkers = 1; - std::vector> m_workers; - std::shared_ptr m_sync_worker; - std::vector> m_source_info_list; - bool m_inprogress = false; - map> m_inprogress_infos; - std::shared_ptr m_cur_source_info; - -friend class sinsp_pep_flt_worker; -}; diff --git a/userspace/libsinsp/settings.h b/userspace/libsinsp/settings.h index 4242034c6..ba278fa7b 100644 --- a/userspace/libsinsp/settings.h +++ b/userspace/libsinsp/settings.h @@ -111,9 +111,3 @@ static const unsigned MAX_JSON_SIZE_B = 500 * 1024; // 500 kiB #define K8S_DATA_MAX_B 100 * 1024 * 1024 #define K8S_DATA_CHUNK_WAIT_US 1000 #define METADATA_DATA_WATCH_FREQ_SEC 1 - -// -// Define this to enable parallel (multi-threaded) filtering of events coming -// from plugins -// -#undef MULTITHREAD_PLUGIN_EVT_PROCESSOR_ENABLED diff --git a/userspace/libsinsp/sinsp.cpp b/userspace/libsinsp/sinsp.cpp index 9705d1485..f1ca0ffb7 100644 --- a/userspace/libsinsp/sinsp.cpp +++ b/userspace/libsinsp/sinsp.cpp @@ -36,7 +36,6 @@ limitations under the License. #include "protodecoder.h" #include "dns_manager.h" #include "plugin.h" -#include "plugin_evt_processor.h" #ifndef CYGWING_AGENT @@ -78,14 +77,12 @@ sinsp::sinsp(bool static_container, const std::string static_id, const std::stri #endif m_h = NULL; m_parser = NULL; - m_plugin_evt_processor = NULL; m_dumper = NULL; m_is_dumping = false; m_metaevt = NULL; m_meinfo.m_piscapevt = NULL; m_network_interfaces = NULL; m_parser = new sinsp_parser(this); - m_plugin_evt_processor = new sinsp_plugin_evt_processor(this); m_thread_manager = new sinsp_thread_manager(this); m_max_fdtable_size = MAX_FD_TABLE_SIZE; m_inactive_container_scan_time_ns = DEFAULT_INACTIVE_CONTAINER_SCAN_TIME_S * ONE_SECOND_IN_NS; @@ -186,12 +183,6 @@ sinsp::~sinsp() m_parser = NULL; } - if(m_plugin_evt_processor) - { - delete m_plugin_evt_processor; - m_plugin_evt_processor = NULL; - } - if(m_thread_manager) { delete m_thread_manager; @@ -1126,9 +1117,6 @@ int32_t sinsp::next(OUT sinsp_evt **puevt) { sinsp_evt* evt; int32_t res; -#ifdef MULTITHREAD_PLUGIN_EVT_PROCESSOR_ENABLED - bool from_plugin_proc_backlog = false; -#endif // // Check if there are fake cpu events to events @@ -1153,69 +1141,58 @@ int32_t sinsp::next(OUT sinsp_evt **puevt) #endif else { -#ifdef MULTITHREAD_PLUGIN_EVT_PROCESSOR_ENABLED - evt = m_plugin_evt_processor->get_event_from_backlog(); - if(evt != NULL) - { - res = SCAP_SUCCESS; - from_plugin_proc_backlog = true; - } - else -#endif - { - evt = &m_evt; + evt = &m_evt; - // - // Reset previous event's decoders if required - // - if(m_decoders_reset_list.size() != 0) + // + // Reset previous event's decoders if required + // + if(m_decoders_reset_list.size() != 0) + { + vector::iterator it; + for(it = m_decoders_reset_list.begin(); it != m_decoders_reset_list.end(); ++it) { - vector::iterator it; - for(it = m_decoders_reset_list.begin(); it != m_decoders_reset_list.end(); ++it) - { - (*it)->on_reset(evt); - } - - m_decoders_reset_list.clear(); + (*it)->on_reset(evt); } - // - // Get the event from libscap - // - res = scap_next(m_h, &(evt->m_pevt), &(evt->m_cpuid)); + m_decoders_reset_list.clear(); + } - if(res != SCAP_SUCCESS) + // + // Get the event from libscap + // + res = scap_next(m_h, &(evt->m_pevt), &(evt->m_cpuid)); + + if(res != SCAP_SUCCESS) + { + if(res == SCAP_TIMEOUT) { - if(res == SCAP_TIMEOUT) - { - if (m_external_event_processor) - { - m_external_event_processor->process_event(NULL, libsinsp::EVENT_RETURN_TIMEOUT); - } - *puevt = NULL; - return res; - } - else if(res == SCAP_EOF) - { - if (m_external_event_processor) - { - m_external_event_processor->process_event(NULL, libsinsp::EVENT_RETURN_EOF); - } - } - else if(res == SCAP_UNEXPECTED_BLOCK) + if (m_external_event_processor) { - uint64_t filepos = scap_ftell(m_h) - scap_get_unexpected_block_readsize(m_h); - restart_capture_at_filepos(filepos); - return SCAP_TIMEOUT; - + m_external_event_processor->process_event(NULL, libsinsp::EVENT_RETURN_TIMEOUT); } - else + *puevt = NULL; + return res; + } + else if(res == SCAP_EOF) + { + if (m_external_event_processor) { - m_lasterr = scap_getlasterr(m_h); + m_external_event_processor->process_event(NULL, libsinsp::EVENT_RETURN_EOF); } + } + else if(res == SCAP_UNEXPECTED_BLOCK) + { + uint64_t filepos = scap_ftell(m_h) - scap_get_unexpected_block_readsize(m_h); + restart_capture_at_filepos(filepos); + return SCAP_TIMEOUT; - return res; } + else + { + m_lasterr = scap_getlasterr(m_h); + } + + return res; } } @@ -1355,24 +1332,7 @@ int32_t sinsp::next(OUT sinsp_evt **puevt) return SCAP_TIMEOUT; } #else - if(evt->get_type() == PPME_PLUGINEVENT_E) - { -#ifdef MULTITHREAD_PLUGIN_EVT_PROCESSOR_ENABLED - if(!from_plugin_proc_backlog) -#endif // MULTITHREAD_PLUGIN_EVT_PROCESSOR_ENABLED - { - evt = m_plugin_evt_processor->process_event(evt); - if(evt == NULL) - { - *puevt = evt; - return SCAP_TIMEOUT; - } - } - } - else - { - m_parser->process_event(evt); - } + m_parser->process_event(evt); #endif // @@ -1794,8 +1754,6 @@ void sinsp::set_filter(const string& filter) sinsp_filter_compiler compiler(this, filter); m_filter = compiler.compile(); m_filterstring = filter; - - m_plugin_evt_processor->compile(filter); } const string sinsp::get_filter() diff --git a/userspace/libsinsp/sinsp.h b/userspace/libsinsp/sinsp.h index 7f2a24f72..5e439a69e 100644 --- a/userspace/libsinsp/sinsp.h +++ b/userspace/libsinsp/sinsp.h @@ -113,7 +113,6 @@ class k8s; class sinsp_partial_tracer; class mesos; class sinsp_plugin; -class sinsp_plugin_evt_processor; #if defined(HAS_CAPTURE) && !defined(_WIN32) class sinsp_ssl; @@ -943,10 +942,6 @@ class SINSP_PUBLIC sinsp : public capture_stats_source, public wmi_handle_source const std::vector>& get_plugins(); std::shared_ptr get_plugin_by_id(uint32_t plugin_id); std::shared_ptr get_source_plugin_by_source(const std::string &source); - sinsp_plugin_evt_processor* get_plugin_evt_processor() - { - return m_plugin_evt_processor; - } uint64_t get_lastevent_ts() const { return m_lastevent_ts; } @@ -1053,7 +1048,6 @@ VISIBILITY_PRIVATE uint64_t m_lastevent_ts; // the parsing engine sinsp_parser* m_parser; - sinsp_plugin_evt_processor* m_plugin_evt_processor; // the statistics analysis engine scap_dumper_t* m_dumper; bool m_is_dumping; From 2cd59ea179790f75b165085c4bee1d47b11d6b54 Mon Sep 17 00:00:00 2001 From: Mark Stemm Date: Fri, 8 Oct 2021 12:10:21 -0700 Subject: [PATCH 032/148] Add a plugin API function to free allocated memory Add a plugin API function to free any memory allocated by the plugin. The dynamic symbol is plugin_free_mem(). In create_plugin(), try to resolve plugin_free_mem first and return an error if it can't be found. Otherwise, save it in the plugin_info structs. In other places, where the framework was calling free() call the resolved symbol instead. Signed-off-by: Mark Stemm --- userspace/libscap/plugin_info.h | 23 ++++++++++++++++++----- userspace/libscap/scap.c | 18 +++++++++--------- userspace/libsinsp/plugin.cpp | 19 +++++++++++++++++-- userspace/libsinsp/plugin.h | 5 +++-- 4 files changed, 47 insertions(+), 18 deletions(-) diff --git a/userspace/libscap/plugin_info.h b/userspace/libscap/plugin_info.h index 57dbf0a74..f4d431282 100644 --- a/userspace/libscap/plugin_info.h +++ b/userspace/libscap/plugin_info.h @@ -81,7 +81,7 @@ typedef enum ss_plugin_field_type // - data: pointer to a memory buffer pointer. The plugin will set it // to point to the memory containing the next event. Once returned, // the memory is owned by the plugin framework and will be freed via -// a call to free(). +// a call to plugin_free_mem(). // - datalen: pointer to a 32bit integer. The plugin will set it the size of the // buffer pointed by data. // - ts: the event timestamp, in nanoseconds since the epoch. @@ -115,7 +115,8 @@ typedef struct ss_plugin_event // extracted value for the provided field, false otherwise // - res_str: if the corresponding field was type==string, this should be // filled in with the string value. The string should be allocated by -// the plugin using malloc() and will be free()d by the plugin framework. +// the plugin using malloc()/similar and will be free()d by the plugin +// framework by calling plugin_free_mem(). // - res_u64: if the corresponding field was type==uint64, this should be // filled in with the uint64 value. @@ -175,7 +176,7 @@ typedef void ss_instance_t; // // NOTE: For all functions below that return a char */struct *, the memory // pointed to by the char */struct * must be allocated by the plugin using -// malloc() and should be freed by the caller using free(). +// malloc()/similar and should be freed by the caller using plugin_free_mem(). // // @@ -195,6 +196,12 @@ typedef struct // char* (*get_required_api_version)(); // + // The plugin framework will call this function to free any + // memory allocated by the plugin and returned to the + // framework. This includes return values from get_type()/get_name()/..., + // get_last_error(), event structs returned in next_batch(), etc. + void (*free_mem)(void *ptr); + // // Return the plugin type. // Required: yes // Should return TYPE_SOURCE_PLUGIN. It still makes sense to @@ -330,7 +337,7 @@ typedef struct // allocate a ss_plugin_event struct using malloc(), as well as // allocate the data buffer within the ss_plugin_event struct. // Both the struct and data buffer are owned by the plugin framework - // and will free them using free(). + // and will free them using plugin_free_mem(). // // Return value: the status of the operation (e.g. SS_PLUGIN_SUCCESS=0, SS_PLUGIN_FAILURE=1, // SS_PLUGIN_TIMEOUT=-1) @@ -384,7 +391,7 @@ typedef struct // allocate an array of contiguous ss_plugin_event structs using malloc(), // as well as allocate each data buffer within each ss_plugin_event // struct using malloc(). Both the array of structs and each data buffer are - // owned by the plugin framework and will free them using free(). + // owned by the plugin framework and will free them using plugin_free_mem(). // Required: no // int32_t (*next_batch)(ss_plugin_t* s, ss_instance_t* h, uint32_t *nevts, ss_plugin_event **evts); @@ -420,6 +427,12 @@ typedef struct // char* (*get_required_api_version)(); // + // The plugin framework will call this function to free any + // memory allocated by the plugin and returned to the + // framework. This includes return values from get_type()/get_name()/..., + // get_last_error(), strings in extract_fields(), etc. + void (*free_mem)(void *ptr); + // // Return the plugin type. // Required: yes // Should return TYPE_EXTRACTOR_PLUGIN. It still makes sense to diff --git a/userspace/libscap/scap.c b/userspace/libscap/scap.c index dfe3b6380..75ffd44e2 100644 --- a/userspace/libscap/scap.c +++ b/userspace/libscap/scap.c @@ -90,13 +90,13 @@ static void scap_free_plugin_batch_state(scap_t* handle) { for(uint32_t i = 0; i < handle->m_input_plugin_batch_nevts; i++) { - free(handle->m_input_plugin_batch_evts[i].data); + handle->m_input_plugin->free_mem(handle->m_input_plugin_batch_evts[i].data); } handle->m_input_plugin_batch_nevts = 0; handle->m_input_plugin_batch_idx = 0; - free(handle->m_input_plugin_batch_evts); + handle->m_input_plugin->free_mem(handle->m_input_plugin_batch_evts); handle->m_input_plugin_batch_evts = NULL; } @@ -1002,7 +1002,7 @@ scap_t* scap_open_plugin_int(char *error, int32_t *rc, source_plugin_info* input { char *errstr = handle->m_input_plugin->get_last_error(handle->m_input_plugin->state); snprintf(error, SCAP_LASTERR_SIZE, "%s", errstr); - free(errstr); + handle->m_input_plugin->free_mem(errstr); scap_close(handle); return NULL; } @@ -1178,7 +1178,7 @@ void scap_close(scap_t* handle) handle->m_input_plugin->close(handle->m_input_plugin->state, handle->m_input_plugin->handle); scap_free_plugin_batch_state(handle); // name was allocated - free(handle->m_input_plugin->name); + handle->m_input_plugin->free_mem(handle->m_input_plugin->name); } #if CYGWING_AGENT || _WIN32 @@ -1723,7 +1723,7 @@ static int32_t scap_next_plugin(scap_t* handle, OUT scap_evt** pevent, OUT uint1 { char *errstr = handle->m_input_plugin->get_last_error(handle->m_input_plugin->state); snprintf(handle->m_lasterr, SCAP_LASTERR_SIZE, "%s", errstr); - free(errstr); + handle->m_input_plugin->free_mem(errstr); } int32_t tres = handle->m_input_plugin_last_batch_res; handle->m_input_plugin_last_batch_res = SCAP_SUCCESS; @@ -1749,7 +1749,7 @@ static int32_t scap_next_plugin(scap_t* handle, OUT scap_evt** pevent, OUT uint1 { char *errstr = handle->m_input_plugin->get_last_error(handle->m_input_plugin->state); snprintf(handle->m_lasterr, SCAP_LASTERR_SIZE, "%s", errstr); - free(errstr); + handle->m_input_plugin->free_mem(errstr); } return handle->m_input_plugin_last_batch_res; } @@ -1778,7 +1778,7 @@ static int32_t scap_next_plugin(scap_t* handle, OUT scap_evt** pevent, OUT uint1 { char *errstr = handle->m_input_plugin->get_last_error(handle->m_input_plugin->state); snprintf(handle->m_lasterr, SCAP_LASTERR_SIZE, "%s", errstr); - free(errstr); + handle->m_input_plugin->free_mem(errstr); } return res; } @@ -1815,9 +1815,9 @@ static int32_t scap_next_plugin(scap_t* handle, OUT scap_evt** pevent, OUT uint1 if(should_free_plugin_evt) { - free(plugin_evt->data); + handle->m_input_plugin->free_mem(plugin_evt->data); plugin_evt->data = NULL; - free(plugin_evt); + handle->m_input_plugin->free_mem(plugin_evt); } if(plugin_evt->ts != UINT64_MAX) diff --git a/userspace/libsinsp/plugin.cpp b/userspace/libsinsp/plugin.cpp index 707544394..07056127f 100755 --- a/userspace/libsinsp/plugin.cpp +++ b/userspace/libsinsp/plugin.cpp @@ -397,6 +397,16 @@ std::shared_ptr sinsp_plugin::create_plugin(string &filepath, char return ret; } + // Get the plugin's free() function, and return an error if it + // doesn't exist. + void (*free_mem)(void *ptr); + *(void **) (&free_mem) = getsym(handle, "plugin_free_mem", errstr); + if(free_mem == NULL) + { + errstr = string("Could not resolve plugin_free_mem function"); + return ret; + } + // Before doing anything else, check the required api // version. If it doesn't match, return an error. @@ -411,7 +421,9 @@ std::shared_ptr sinsp_plugin::create_plugin(string &filepath, char return ret; } - std::string version_str = str_from_alloc_charbuf(get_required_api_version()); + char *version_cstr = get_required_api_version(); + std::string version_str = version_cstr; + free_mem(version_cstr); version v(version_str); if(!v.m_valid) { @@ -671,7 +683,7 @@ std::string sinsp_plugin::str_from_alloc_charbuf(char *charbuf) if(charbuf != NULL) { str = charbuf; - free(charbuf); + m_plugin_info.free_mem(charbuf); } return str; @@ -681,6 +693,7 @@ bool sinsp_plugin::resolve_dylib_symbols(void *handle, std::string &errstr) { // Some functions are required and return false if not found. if((*(void **) (&(m_plugin_info.get_required_api_version)) = getsym(handle, "plugin_get_required_api_version", errstr)) == NULL || + (*(void **) (&(m_plugin_info.free_mem)) = getsym(handle, "plugin_free_mem", errstr)) == NULL || (*(void **) (&(m_plugin_info.get_last_error)) = getsym(handle, "plugin_get_last_error", errstr)) == NULL || (*(void **) (&(m_plugin_info.get_name)) = getsym(handle, "plugin_get_name", errstr)) == NULL || (*(void **) (&(m_plugin_info.get_description)) = getsym(handle, "plugin_get_description", errstr)) == NULL || @@ -923,6 +936,7 @@ bool sinsp_source_plugin::resolve_dylib_symbols(void *handle, std::string &errst // // Some functions are required and return false if not found. if((*(void **) (&(m_source_plugin_info.get_required_api_version)) = getsym(handle, "plugin_get_required_api_version", errstr)) == NULL || + (*(void **) (&(m_source_plugin_info.free_mem)) = getsym(handle, "plugin_free_mem", errstr)) == NULL || (*(void **) (&(m_source_plugin_info.init)) = getsym(handle, "plugin_init", errstr)) == NULL || (*(void **) (&(m_source_plugin_info.destroy)) = getsym(handle, "plugin_destroy", errstr)) == NULL || (*(void **) (&(m_source_plugin_info.get_last_error)) = getsym(handle, "plugin_get_last_error", errstr)) == NULL || @@ -1000,6 +1014,7 @@ bool sinsp_extractor_plugin::resolve_dylib_symbols(void *handle, std::string &er // // Some functions are required and return false if not found. if((*(void **) (&(m_extractor_plugin_info.get_required_api_version)) = getsym(handle, "plugin_get_required_api_version", errstr)) == NULL || + (*(void **) (&(m_extractor_plugin_info.free_mem)) = getsym(handle, "plugin_free_mem", errstr)) == NULL || (*(void **) (&(m_extractor_plugin_info.init)) = getsym(handle, "plugin_init", errstr)) == NULL || (*(void **) (&(m_extractor_plugin_info.destroy)) = getsym(handle, "plugin_destroy", errstr)) == NULL || (*(void **) (&(m_extractor_plugin_info.get_last_error)) = getsym(handle, "plugin_get_last_error", errstr)) == NULL || diff --git a/userspace/libsinsp/plugin.h b/userspace/libsinsp/plugin.h index 57d3fe3d0..3ce4c934a 100755 --- a/userspace/libsinsp/plugin.h +++ b/userspace/libsinsp/plugin.h @@ -70,7 +70,7 @@ class sinsp_async_extractor SHUTDOWN_DONE = 5, }; - // On success the caller is responsible for free() in res_str + // On success the caller is responsible for plugin_free() in res_str // when ftype == PT_CHARBUF. inline int32_t extract(ss_plugin_event &evt, ss_plugin_extract_field &field) { @@ -258,7 +258,7 @@ class sinsp_plugin static void* getsym(void* handle, const char* name, std::string &errstr); // Helper function to set a string from an allocated charbuf and free the charbuf. - static std::string str_from_alloc_charbuf(char *charbuf); + std::string str_from_alloc_charbuf(char *charbuf); // init() will call this to save the resulting state struct virtual void set_plugin_state(ss_plugin_t *state) = 0; @@ -270,6 +270,7 @@ class sinsp_plugin // included here as they are called in create_plugin() typedef struct { char* (*get_required_api_version)(); + void (*free_mem)(void *ptr); ss_plugin_t* (*init)(char* config, int32_t* rc); void (*destroy)(ss_plugin_t* s); char* (*get_last_error)(ss_plugin_t* s); From 97c240636b6d1715eb23b4d4c65b571b7cad7a59 Mon Sep 17 00:00:00 2001 From: Mark Stemm Date: Fri, 8 Oct 2021 16:31:00 -0700 Subject: [PATCH 033/148] Move async extraction from plugin framework to plugin-sdk-go Move the handling of async extraction from the plugin framework to plugin-sdk-go. All go plugins now export a plugin_extract_fields function in C. That function either uses the async extraction interface implemented in plugin-sdk-go, or calls the (synchronous) extraction functions. Signed-off-by: Mark Stemm --- userspace/libscap/plugin_info.h | 25 +----- userspace/libsinsp/plugin.cpp | 36 ++------ userspace/libsinsp/plugin.h | 151 +------------------------------- userspace/libsinsp/sinsp.cpp | 10 --- userspace/libsinsp/sinsp.h | 5 -- 5 files changed, 12 insertions(+), 215 deletions(-) diff --git a/userspace/libscap/plugin_info.h b/userspace/libscap/plugin_info.h index f4d431282..c26d8c3e7 100644 --- a/userspace/libscap/plugin_info.h +++ b/userspace/libscap/plugin_info.h @@ -60,9 +60,9 @@ typedef enum ss_plugin_field_type }ss_plugin_field_type; // Values to return from init() / open() / next() / next_batch() / -// extract_fields() / register_async_extractor(). Note that these -// values map exactly to the corresponding SCAP_XXX values in scap.h, -// and should be kept in sync. +// extract_fields(). Note that these values map exactly to the +// corresponding SCAP_XXX values in scap.h, and should be kept in +// sync. #define SS_PLUGIN_SUCCESS 0 #define SS_PLUGIN_FAILURE 1 #define SS_PLUGIN_TIMEOUT -1 @@ -131,20 +131,6 @@ typedef struct ss_plugin_extract_field uint64_t res_u64; } ss_plugin_extract_field; -// Used by the async extraction interface -typedef bool (*cb_wait_t)(void* wait_ctx); - -typedef struct async_extractor_info -{ - // Pointer as this allows swapping out events from other - // structs. - const ss_plugin_event *evt; - ss_plugin_extract_field *field; - int32_t rc; - cb_wait_t cb_wait; - void* wait_ctx; -} async_extractor_info; - // // This is the opaque pointer to the state of a plugin. // It points to any data that might be needed plugin-wise. It is @@ -395,11 +381,6 @@ typedef struct // Required: no // int32_t (*next_batch)(ss_plugin_t* s, ss_instance_t* h, uint32_t *nevts, ss_plugin_event **evts); - // - // This is an optional, internal, function used to speed up value extraction - // Required: no - // - int32_t (*register_async_extractor)(ss_plugin_t *s, async_extractor_info *info); // // The following members are PRIVATE for the engine and should not be touched. diff --git a/userspace/libsinsp/plugin.cpp b/userspace/libsinsp/plugin.cpp index 07056127f..a45ccd3ae 100755 --- a/userspace/libsinsp/plugin.cpp +++ b/userspace/libsinsp/plugin.cpp @@ -351,10 +351,10 @@ std::string sinsp_plugin::version::as_string() const std::to_string(m_version_patch); } -std::shared_ptr sinsp_plugin::register_plugin(sinsp* inspector, string filepath, char* config, bool avoid_async) +std::shared_ptr sinsp_plugin::register_plugin(sinsp* inspector, string filepath, char* config) { string errstr; - std::shared_ptr plugin = create_plugin(filepath, config, avoid_async, errstr); + std::shared_ptr plugin = create_plugin(filepath, config, errstr); if (!plugin) { @@ -382,7 +382,7 @@ std::shared_ptr sinsp_plugin::register_plugin(sinsp* inspector, st return plugin; } -std::shared_ptr sinsp_plugin::create_plugin(string &filepath, char* config, bool avoid_async, std::string &errstr) +std::shared_ptr sinsp_plugin::create_plugin(string &filepath, char* config, std::string &errstr) { std::shared_ptr ret; @@ -475,7 +475,7 @@ std::shared_ptr sinsp_plugin::create_plugin(string &filepath, char errstr = ""; // Initialize the plugin - if (!ret->init(config, avoid_async)) + if (!ret->init(config)) { ret = NULL; } @@ -516,7 +516,7 @@ sinsp_plugin::~sinsp_plugin() { } -bool sinsp_plugin::init(char *config, bool avoid_async) +bool sinsp_plugin::init(char *config) { if (!m_plugin_info.init) { @@ -535,16 +535,6 @@ bool sinsp_plugin::init(char *config, bool avoid_async) set_plugin_state(state); - if(m_plugin_info.register_async_extractor && !avoid_async) - { - m_async_extractor.reset(new sinsp_async_extractor()); - - if(m_plugin_info.register_async_extractor(plugin_state(), m_async_extractor->extractor_info()) != SCAP_SUCCESS) - { - throw sinsp_exception(string("error in plugin ") + m_name + ": " + get_last_error()); - } - } - return true; } @@ -623,14 +613,7 @@ bool sinsp_plugin::extract_field(ss_plugin_event &evt, sinsp_plugin::ext_field & int32_t rc; - if(m_async_extractor) - { - rc = m_async_extractor->extract_field(evt, efield); - } - else - { - rc = m_plugin_info.extract_fields(plugin_state(), &evt, num_fields, &efield); - } + rc = m_plugin_info.extract_fields(plugin_state(), &evt, num_fields, &efield); if (rc != SCAP_SUCCESS) { @@ -708,7 +691,6 @@ bool sinsp_plugin::resolve_dylib_symbols(void *handle, std::string &errstr) (*(void **) (&m_plugin_info.destroy)) = getsym(handle, "plugin_destroy", errstr); (*(void **) (&m_plugin_info.get_fields)) = getsym(handle, "plugin_get_fields", errstr); (*(void **) (&m_plugin_info.extract_fields)) = getsym(handle, "plugin_extract_fields", errstr); - (*(void **) (&m_plugin_info.register_async_extractor)) = getsym(handle, "plugin_register_async_extractor", errstr); m_name = str_from_alloc_charbuf(m_plugin_info.get_name()); m_description = str_from_alloc_charbuf(m_plugin_info.get_description()); @@ -823,11 +805,6 @@ bool sinsp_plugin::resolve_dylib_symbols(void *handle, std::string &errstr) return true; } -void sinsp_plugin::disable_async_extract() -{ - m_async_extractor.reset(); -} - sinsp_source_plugin::sinsp_source_plugin() { memset(&m_source_plugin_info, 0, sizeof(m_source_plugin_info)); @@ -961,7 +938,6 @@ bool sinsp_source_plugin::resolve_dylib_symbols(void *handle, std::string &errst (*(void **) (&m_source_plugin_info.event_to_string)) = getsym(handle, "plugin_event_to_string", errstr); (*(void **) (&m_source_plugin_info.extract_fields)) = getsym(handle, "plugin_extract_fields", errstr); (*(void **) (&m_source_plugin_info.next_batch)) = getsym(handle, "plugin_next_batch", errstr); - (*(void **) (&m_source_plugin_info.register_async_extractor)) = getsym(handle, "plugin_register_async_extractor", errstr); m_id = m_source_plugin_info.get_id(); m_event_source = str_from_alloc_charbuf(m_source_plugin_info.get_event_source()); diff --git a/userspace/libsinsp/plugin.h b/userspace/libsinsp/plugin.h index 3ce4c934a..1dc438682 100755 --- a/userspace/libsinsp/plugin.h +++ b/userspace/libsinsp/plugin.h @@ -28,145 +28,6 @@ limitations under the License. class sinsp_filter_check_plugin; -class sinsp_async_extractor -{ -public: - sinsp_async_extractor() - { - m_lock = state::INIT; - - m_async_extractor_info.cb_wait = [](void *wait_ctx) - { - return static_cast(wait_ctx)->wait(); - }; - - m_async_extractor_info.wait_ctx = this; - } - - ~sinsp_async_extractor() - { - shutdown(); - } - - struct async_extractor_info *extractor_info() - { - return &m_async_extractor_info; - } - - int32_t extract_field(ss_plugin_event &evt, ss_plugin_extract_field &field) - { - return extract(evt, field); - } - -private: - - enum state - { - INIT = 0, - INPUT_READY = 1, - PROCESSING = 2, - DONE = 3, - SHUTDOWN_REQ = 4, - SHUTDOWN_DONE = 5, - }; - - // On success the caller is responsible for plugin_free() in res_str - // when ftype == PT_CHARBUF. - inline int32_t extract(ss_plugin_event &evt, ss_plugin_extract_field &field) - { - m_async_extractor_info.evt = &evt; - m_async_extractor_info.field = &field; - - int old_val = state::DONE; - - while(!m_lock.compare_exchange_strong(old_val, state::INPUT_READY)) - { - old_val = state::DONE; - } - - // - // Once INPUT_READY state has been aquired, wait for worker completition - // - while(m_lock != state::DONE); - - // rc now contains the error code for the extraction. - return m_async_extractor_info.rc; - } - - inline bool wait() - { - m_lock = state::DONE; - uint64_t ncycles = 0; - bool sleeping = false; - - // - // Worker has done and now waits for a new input or a shutdown request. - // Note: we busy loop for the first 1ms to guarantee maximum performance. - // After 1ms we start sleeping to conserve CPU. - // - int old_val = state::INPUT_READY; - - auto start_time = chrono::high_resolution_clock::now(); - - while(!m_lock.compare_exchange_strong(old_val, state::PROCESSING)) - { - // shutdown - if(old_val == state::SHUTDOWN_REQ) - { - m_lock = state::SHUTDOWN_DONE; - return false; - } - old_val = state::INPUT_READY; - - if(sleeping) - { - this_thread::sleep_for(chrono::milliseconds(10)); - } - else - { - ncycles++; - if(ncycles >= 100000) - { - auto cur_time = chrono::high_resolution_clock::now(); - auto delta_time = chrono::duration_cast(cur_time - start_time).count(); - if(delta_time > 1000) - { - sleeping = true; - } - else - { - ncycles = 0; - } - } - } - } - return true; - } - - inline void shutdown() - { - // - // Set SHUTDOWN_REQ iff the worker - // - int old_val = state::DONE; - while(m_lock.compare_exchange_strong(old_val, state::SHUTDOWN_REQ)) - { - old_val = state::DONE; - } - - // await shutdown - // XXX/mstemm add a timeout to this - while (m_lock != state::SHUTDOWN_DONE) - ; - } - -private: - // The shared struct to communicate with the plugin side - struct async_extractor_info m_async_extractor_info; - - atomic m_lock; -}; - // Base class for source/extractor plugins. Can not be created directly. class sinsp_plugin { @@ -214,12 +75,12 @@ class sinsp_plugin // Create and register a plugin from a shared library pointed // to by filepath, and add it to the inspector. // The created sinsp_plugin is returned. - static std::shared_ptr register_plugin(sinsp* inspector, std::string filepath, char *config, bool avoid_async); + static std::shared_ptr register_plugin(sinsp* inspector, std::string filepath, char *config); // Create a plugin from the dynamic library at the provided // path. On error, the shared_ptr will == NULL and errstr is // set with an error. - static std::shared_ptr create_plugin(std::string &filepath, char *config, bool avoid_async, std::string &errstr); + static std::shared_ptr create_plugin(std::string &filepath, char *config, std::string &errstr); // Return a string with names/descriptions/etc of all plugins used by this inspector static std::list plugin_infos(sinsp *inspector); @@ -233,9 +94,7 @@ class sinsp_plugin // Returns true on success, false + sets errstr on error. virtual bool resolve_dylib_symbols(void *handle, std::string &errstr); - void disable_async_extract(); - - bool init(char *config, bool avoid_async); + bool init(char *config); void destroy(); virtual ss_plugin_type type() = 0; @@ -250,7 +109,6 @@ class sinsp_plugin const filtercheck_field_info *fields(); uint32_t nfields(); - // Will either use the the async extractor interface or direct C calls to extract the values bool extract_field(ss_plugin_event &evt, sinsp_plugin::ext_field &field); protected: @@ -280,7 +138,6 @@ class sinsp_plugin char* (*get_version)(); char* (*get_fields)(); int32_t (*extract_fields)(ss_plugin_t *s, const ss_plugin_event *evt, uint32_t num_fields, ss_plugin_extract_field *fields); - int32_t (*register_async_extractor)(ss_plugin_t *s, async_extractor_info *info); } common_plugin_info; std::string m_name; @@ -293,8 +150,6 @@ class sinsp_plugin std::unique_ptr m_fields; int32_t m_nfields; - std::unique_ptr m_async_extractor; - common_plugin_info m_plugin_info; }; diff --git a/userspace/libsinsp/sinsp.cpp b/userspace/libsinsp/sinsp.cpp index f1ca0ffb7..b6e4fc869 100644 --- a/userspace/libsinsp/sinsp.cpp +++ b/userspace/libsinsp/sinsp.cpp @@ -164,8 +164,6 @@ sinsp::sinsp(bool static_container, const std::string static_id, const std::stri #endif // !defined(CYGWING_AGENT) && !defined(MINIMAL_BUILD) m_filter_proc_table_when_saving = false; - - m_n_async_plugin_extractors = 0; } sinsp::~sinsp() @@ -1605,14 +1603,6 @@ void sinsp::set_statsd_port(const uint16_t port) void sinsp::add_plugin(std::shared_ptr plugin) { - uint32_t ncpus = thread::hardware_concurrency(); - bool avoid_async = ncpus == 0 || (m_n_async_plugin_extractors >= (ncpus - 1)); - - if(avoid_async) - { - plugin->disable_async_extract(); - } - for(auto& it : m_plugins_list) { if(it->name() == plugin->name()) diff --git a/userspace/libsinsp/sinsp.h b/userspace/libsinsp/sinsp.h index 5e439a69e..e13363a30 100644 --- a/userspace/libsinsp/sinsp.h +++ b/userspace/libsinsp/sinsp.h @@ -1236,11 +1236,6 @@ VISIBILITY_PRIVATE // std::vector> m_plugins_list; // - // Count of plugins that are using a full CPU to accelerate field - // extraction. We want this to be lower than the number of available CPUs. - // - uint32_t m_n_async_plugin_extractors; - // // The ID of the plugin to use as event input, or zero // if no source plugin should be used as source // From 92a8e2e24b0d6f5d49360a70f6e7dd9c7cd74b37 Mon Sep 17 00:00:00 2001 From: Mark Stemm Date: Thu, 14 Oct 2021 15:45:22 -0700 Subject: [PATCH 034/148] Allow for alt. list of filterchecks when creating filter/formatter When creating filter/formatter factories, or when creating an individual filter/formatter, allow specifying an alternate list of available filterchecks. This will allow segregating fields used by plugins, which only work with a given plugin event, from fields used by syscalls, which work with the "default" set of filterchecks in g_filterlist. The sinsp_filter_check_list object is a good match for this, but it represents both a list of filter_checks and also has the default set of filterchecks for syscalls baked in. With the notion of factories, we need the first part but don't want the second. So make up a base class filter_check_list that has the first part. sinsp_filter_check_list derives from the base class and has the built-in filterchecks. Also move it to its own file so it's possible to include the declaration without including filterchecks.h. Signed-off-by: Mark Stemm --- userspace/libsinsp/CMakeLists.txt | 1 + userspace/libsinsp/eventformatter.cpp | 25 +++-- userspace/libsinsp/eventformatter.h | 11 +- userspace/libsinsp/fields_info.cpp | 4 +- userspace/libsinsp/filter.cpp | 117 ++------------------ userspace/libsinsp/filter.h | 5 +- userspace/libsinsp/filter_check_list.cpp | 133 +++++++++++++++++++++++ userspace/libsinsp/filter_check_list.h | 58 ++++++++++ userspace/libsinsp/filterchecks.h | 20 +--- userspace/libsinsp/plugin.cpp | 11 +- userspace/libsinsp/plugin.h | 9 +- userspace/libsinsp/sinsp.cpp | 4 +- userspace/libsinsp/sinsp.h | 2 +- userspace/libsinsp/table.cpp | 2 +- userspace/libsinsp/utils.cpp | 3 +- userspace/libsinsp/utils.h | 2 +- 16 files changed, 250 insertions(+), 157 deletions(-) create mode 100644 userspace/libsinsp/filter_check_list.cpp create mode 100644 userspace/libsinsp/filter_check_list.h diff --git a/userspace/libsinsp/CMakeLists.txt b/userspace/libsinsp/CMakeLists.txt index cd7fc2820..2eff18f0c 100644 --- a/userspace/libsinsp/CMakeLists.txt +++ b/userspace/libsinsp/CMakeLists.txt @@ -66,6 +66,7 @@ set(SINSP_SOURCES filter.cpp fields_info.cpp filterchecks.cpp + filter_check_list.cpp gen_filter.cpp http_parser.c http_reason.cpp diff --git a/userspace/libsinsp/eventformatter.cpp b/userspace/libsinsp/eventformatter.cpp index bd702a2ad..d15eac7c5 100644 --- a/userspace/libsinsp/eventformatter.cpp +++ b/userspace/libsinsp/eventformatter.cpp @@ -25,17 +25,20 @@ limitations under the License. // rawstring_check implementation /////////////////////////////////////////////////////////////////////////////// #ifdef HAS_FILTERING -extern sinsp_filter_check_list g_filterlist; -sinsp_evt_formatter::sinsp_evt_formatter(sinsp* inspector) - : m_inspector(inspector) +sinsp_evt_formatter::sinsp_evt_formatter(sinsp* inspector, + filter_check_list &available_checks) + : m_inspector(inspector), + m_available_checks(available_checks) { } -sinsp_evt_formatter::sinsp_evt_formatter(sinsp* inspector, const string& fmt) +sinsp_evt_formatter::sinsp_evt_formatter(sinsp* inspector, + const string& fmt, + filter_check_list &available_checks) + : m_inspector(inspector), + m_available_checks(available_checks) { - m_inspector = inspector; - gen_event_formatter::output_format of = gen_event_formatter::OF_NORMAL; if(m_inspector->get_buffer_format() == sinsp_evt::PF_JSON @@ -145,7 +148,7 @@ void sinsp_evt_formatter::set_format(gen_event_formatter::output_format of, cons } } - sinsp_filter_check* chk = g_filterlist.new_filter_check_from_fldname(string(cfmt + j + 1), + sinsp_filter_check* chk = m_available_checks.new_filter_check_from_fldname(string(cfmt + j + 1), m_inspector, false); @@ -383,8 +386,10 @@ bool sinsp_evt_formatter_cache::tostring(sinsp_evt *evt, string &format, OUT str return get_cached_formatter(format)->tostring(evt, *res); } -sinsp_evt_formatter_factory::sinsp_evt_formatter_factory(sinsp *inspector) - : m_inspector(inspector), m_output_format(gen_event_formatter::OF_NORMAL) +sinsp_evt_formatter_factory::sinsp_evt_formatter_factory(sinsp *inspector, filter_check_list &available_checks) + : m_inspector(inspector), + m_available_checks(available_checks), + m_output_format(gen_event_formatter::OF_NORMAL) { } @@ -410,7 +415,7 @@ std::shared_ptr sinsp_evt_formatter_factory::create_formatt std::shared_ptr ret; - ret.reset(new sinsp_evt_formatter(m_inspector)); + ret.reset(new sinsp_evt_formatter(m_inspector, m_available_checks)); ret->set_format(m_output_format, format); m_formatters[format] = ret; diff --git a/userspace/libsinsp/eventformatter.h b/userspace/libsinsp/eventformatter.h index 3294bf370..af61d55c2 100644 --- a/userspace/libsinsp/eventformatter.h +++ b/userspace/libsinsp/eventformatter.h @@ -21,6 +21,9 @@ limitations under the License. #include #include +#include "filter_check_list.h" +#include "gen_filter.h" + class sinsp_filter_check; /** @defgroup event Event manipulation @@ -44,9 +47,9 @@ class SINSP_PUBLIC sinsp_evt_formatter : public gen_event_formatter as the one of the sysdig '-p' command line flag, so refer to the sysdig manual for details. */ - sinsp_evt_formatter(sinsp* inspector); + sinsp_evt_formatter(sinsp* inspector, filter_check_list &available_checks = g_filterlist); - sinsp_evt_formatter(sinsp* inspector, const string& fmt); + sinsp_evt_formatter(sinsp* inspector, const string& fmt, filter_check_list &available_checks = g_filterlist); void set_format(gen_event_formatter::output_format of, const string& fmt) override; @@ -103,6 +106,7 @@ class SINSP_PUBLIC sinsp_evt_formatter : public gen_event_formatter vector> m_tokens; vector m_tokenlens; sinsp* m_inspector; + filter_check_list &m_available_checks; bool m_require_all_values; vector m_chks_to_free; @@ -145,7 +149,7 @@ class SINSP_PUBLIC sinsp_evt_formatter_cache class sinsp_evt_formatter_factory : public gen_event_formatter_factory { public: - sinsp_evt_formatter_factory(sinsp *inspector); + sinsp_evt_formatter_factory(sinsp *inspector, filter_check_list &available_checks=g_filterlist); virtual ~sinsp_evt_formatter_factory(); void set_output_format(gen_event_formatter::output_format of) override; @@ -158,5 +162,6 @@ class sinsp_evt_formatter_factory : public gen_event_formatter_factory std::map> m_formatters; sinsp *m_inspector; + filter_check_list &m_available_checks; gen_event_formatter::output_format m_output_format; }; diff --git a/userspace/libsinsp/fields_info.cpp b/userspace/libsinsp/fields_info.cpp index 968ed7727..7e5d7840f 100644 --- a/userspace/libsinsp/fields_info.cpp +++ b/userspace/libsinsp/fields_info.cpp @@ -46,7 +46,7 @@ void list_fields(bool verbose, bool markdown, bool names_only) } vector fc_plugins; - sinsp::get_filtercheck_fields_info(&fc_plugins); + sinsp::get_filtercheck_fields_info(fc_plugins); for(j = 0; j < fc_plugins.size(); j++) { @@ -173,4 +173,4 @@ void list_events(sinsp* inspector) printf(")\n"); } -} \ No newline at end of file +} diff --git a/userspace/libsinsp/filter.cpp b/userspace/libsinsp/filter.cpp index 66833de67..a3ce55816 100644 --- a/userspace/libsinsp/filter.cpp +++ b/userspace/libsinsp/filter.cpp @@ -59,110 +59,6 @@ void *memmem(const void *haystack, size_t haystacklen, const void *needle, size_ #endif -extern sinsp_filter_check_list g_filterlist; - -/////////////////////////////////////////////////////////////////////////////// -// sinsp_filter_check_list implementation -/////////////////////////////////////////////////////////////////////////////// -sinsp_filter_check_list::sinsp_filter_check_list() -{ - ////////////////////////////////////////////////////////////////////////////// - // ADD NEW FILTER CHECK CLASSES HERE - ////////////////////////////////////////////////////////////////////////////// - add_filter_check(new sinsp_filter_check_fd()); - add_filter_check(new sinsp_filter_check_thread()); - add_filter_check(new sinsp_filter_check_event()); - add_filter_check(new sinsp_filter_check_user()); - add_filter_check(new sinsp_filter_check_group()); - add_filter_check(new sinsp_filter_check_syslog()); - add_filter_check(new sinsp_filter_check_container()); - add_filter_check(new sinsp_filter_check_utils()); - add_filter_check(new sinsp_filter_check_fdlist()); -#if !defined(CYGWING_AGENT) && !defined(MINIMAL_BUILD) - add_filter_check(new sinsp_filter_check_k8s()); - add_filter_check(new sinsp_filter_check_mesos()); -#endif // !defined(CYGWING_AGENT) && !defined(MINIMAL_BUILD) - add_filter_check(new sinsp_filter_check_tracer()); - add_filter_check(new sinsp_filter_check_evtin()); -} - -sinsp_filter_check_list::~sinsp_filter_check_list() -{ - uint32_t j; - - for(j = 0; j < m_check_list.size(); j++) - { - delete m_check_list[j]; - } -} - -void sinsp_filter_check_list::add_filter_check(sinsp_filter_check* filter_check) -{ - m_check_list.push_back(filter_check); -} - -void sinsp_filter_check_list::get_all_fields(OUT vector* list) -{ - uint32_t j; - - for(j = 0; j < m_check_list.size(); j++) - { - list->push_back((const filter_check_info*)&(m_check_list[j]->m_info)); - } -} - -sinsp_filter_check* sinsp_filter_check_list::new_filter_check_from_fldname(const string& name, - sinsp* inspector, - bool do_exact_check) -{ - uint32_t j; - - for(j = 0; j < m_check_list.size(); j++) - { - m_check_list[j]->m_inspector = inspector; - - int32_t fldnamelen = m_check_list[j]->parse_field_name(name.c_str(), false, true); - - if(fldnamelen != -1) - { - if(do_exact_check) - { - if((int32_t)name.size() != fldnamelen) - { - goto field_not_found; - } - } - - sinsp_filter_check* newchk = m_check_list[j]->allocate_new(); - newchk->set_inspector(inspector); - return newchk; - } - } - -field_not_found: - - // - // If you are implementing a new filter check and this point is reached, - // it's very likely that you've forgotten to add your filter to the list in - // the constructor - // - return NULL; -} - -sinsp_filter_check* sinsp_filter_check_list::new_filter_check_from_another(sinsp_filter_check *chk) -{ - sinsp_filter_check *newchk = chk->allocate_new(); - - newchk->m_inspector = chk->m_inspector; - newchk->m_field_id = chk->m_field_id; - newchk->m_field = &chk->m_info.m_fields[chk->m_field_id]; - - newchk->m_boolop = chk->m_boolop; - newchk->m_cmpop = chk->m_cmpop; - - return newchk; -} - /////////////////////////////////////////////////////////////////////////////// // type-based comparison functions /////////////////////////////////////////////////////////////////////////////// @@ -2430,8 +2326,9 @@ void sinsp_evttype_filter::syscalls_for_ruleset(std::vector &syscalls, uin return m_rulesets[ruleset]->syscalls_for_ruleset(syscalls); } -sinsp_filter_factory::sinsp_filter_factory(sinsp *inspector) - : m_inspector(inspector) +sinsp_filter_factory::sinsp_filter_factory(sinsp *inspector, + filter_check_list &available_checks) + : m_inspector(inspector), m_available_checks(available_checks) { } @@ -2447,9 +2344,9 @@ gen_event_filter *sinsp_filter_factory::new_filter() gen_event_filter_check *sinsp_filter_factory::new_filtercheck(const char *fldname) { - return g_filterlist.new_filter_check_from_fldname(fldname, - m_inspector, - true); + return m_available_checks.new_filter_check_from_fldname(fldname, + m_inspector, + true); } std::list sinsp_filter_factory::get_fields() @@ -2457,7 +2354,7 @@ std::list sinsp_filter_factory std::list ret; vector fc_plugins; - sinsp::get_filtercheck_fields_info(&fc_plugins); + m_available_checks.get_all_fields(fc_plugins); for(auto &fci : fc_plugins) { diff --git a/userspace/libsinsp/filter.h b/userspace/libsinsp/filter.h index 33acb5749..6245627af 100644 --- a/userspace/libsinsp/filter.h +++ b/userspace/libsinsp/filter.h @@ -22,6 +22,7 @@ limitations under the License. #ifdef HAS_FILTERING +#include "filter_check_list.h" #include "gen_filter.h" /** @defgroup filter Filtering events @@ -206,7 +207,8 @@ class SINSP_PUBLIC sinsp_evttype_filter class sinsp_filter_factory : public gen_event_filter_factory { public: - sinsp_filter_factory(sinsp *inspector); + sinsp_filter_factory(sinsp *inspector, filter_check_list &available_checks=g_filterlist); + virtual ~sinsp_filter_factory(); gen_event_filter *new_filter(); @@ -217,6 +219,7 @@ class sinsp_filter_factory : public gen_event_filter_factory protected: sinsp *m_inspector; + filter_check_list &m_available_checks; }; #endif // HAS_FILTERING diff --git a/userspace/libsinsp/filter_check_list.cpp b/userspace/libsinsp/filter_check_list.cpp new file mode 100644 index 000000000..0521d6cee --- /dev/null +++ b/userspace/libsinsp/filter_check_list.cpp @@ -0,0 +1,133 @@ +/* +Copyright (C) 2021 The Falco Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. + +*/ + +#include + +#include "sinsp.h" + +#ifdef HAS_FILTERING +#include "filter_check_list.h" +#include "filterchecks.h" + +using namespace std; + +/////////////////////////////////////////////////////////////////////////////// +// sinsp_filter_check_list implementation +/////////////////////////////////////////////////////////////////////////////// +filter_check_list::filter_check_list() +{ +} + +filter_check_list::~filter_check_list() +{ + for(auto *chk : m_check_list) + { + delete chk; + } +} + +void filter_check_list::add_filter_check(sinsp_filter_check* filter_check) +{ + m_check_list.push_back(filter_check); +} + +void filter_check_list::get_all_fields(vector& list) +{ + for(auto *chk : m_check_list) + { + list.push_back((const filter_check_info*)&(chk->m_info)); + } +} + +sinsp_filter_check* filter_check_list::new_filter_check_from_fldname(const string& name, + sinsp* inspector, + bool do_exact_check) +{ + for(auto *chk : m_check_list) + { + chk->m_inspector = inspector; + + int32_t fldnamelen = chk->parse_field_name(name.c_str(), false, true); + + if(fldnamelen != -1) + { + if(do_exact_check) + { + if((int32_t)name.size() != fldnamelen) + { + goto field_not_found; + } + } + + sinsp_filter_check* newchk = chk->allocate_new(); + newchk->set_inspector(inspector); + return newchk; + } + } + +field_not_found: + + // + // If you are implementing a new filter check and this point is reached, + // it's very likely that you've forgotten to add your filter to the list in + // the constructor + // + return NULL; +} + +sinsp_filter_check* filter_check_list::new_filter_check_from_another(sinsp_filter_check *chk) +{ + sinsp_filter_check *newchk = chk->allocate_new(); + + newchk->m_inspector = chk->m_inspector; + newchk->m_field_id = chk->m_field_id; + newchk->m_field = &chk->m_info.m_fields[chk->m_field_id]; + + newchk->m_boolop = chk->m_boolop; + newchk->m_cmpop = chk->m_cmpop; + + return newchk; +} + +sinsp_filter_check_list::sinsp_filter_check_list() +{ + ////////////////////////////////////////////////////////////////////////////// + // ADD NEW FILTER CHECK CLASSES HERE + ////////////////////////////////////////////////////////////////////////////// + add_filter_check(new sinsp_filter_check_fd()); + add_filter_check(new sinsp_filter_check_thread()); + add_filter_check(new sinsp_filter_check_gen_event()); + add_filter_check(new sinsp_filter_check_event()); + add_filter_check(new sinsp_filter_check_user()); + add_filter_check(new sinsp_filter_check_group()); + add_filter_check(new sinsp_filter_check_syslog()); + add_filter_check(new sinsp_filter_check_container()); + add_filter_check(new sinsp_filter_check_utils()); + add_filter_check(new sinsp_filter_check_fdlist()); +#if !defined(CYGWING_AGENT) && !defined(MINIMAL_BUILD) + add_filter_check(new sinsp_filter_check_k8s()); + add_filter_check(new sinsp_filter_check_mesos()); +#endif // !defined(CYGWING_AGENT) && !defined(MINIMAL_BUILD) + add_filter_check(new sinsp_filter_check_tracer()); + add_filter_check(new sinsp_filter_check_evtin()); +} + +sinsp_filter_check_list::~sinsp_filter_check_list() +{ +} + +#endif // HAS_FILTERING diff --git a/userspace/libsinsp/filter_check_list.h b/userspace/libsinsp/filter_check_list.h new file mode 100644 index 000000000..df0d11cb6 --- /dev/null +++ b/userspace/libsinsp/filter_check_list.h @@ -0,0 +1,58 @@ +/* +Copyright (C) 2021 The Falco Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. + +*/ + +#pragma once + +#include +#include + +#ifdef HAS_FILTERING +class sinsp_filter_check; +class filter_check_info; +class sinsp; + +// +// Global class that stores the list of filtercheck plugins and offers +// functions to work with it. +// +class filter_check_list +{ +public: + filter_check_list(); + ~filter_check_list(); + void add_filter_check(sinsp_filter_check* filter_check); + void get_all_fields(std::vector& list); + sinsp_filter_check* new_filter_check_from_another(sinsp_filter_check *chk); + sinsp_filter_check* new_filter_check_from_fldname(const std::string& name, sinsp* inspector, bool do_exact_check); + +protected: + std::vector m_check_list; +}; + +// +// This bakes in the "default" set of filterchecks that work with syscalls +class sinsp_filter_check_list : public filter_check_list +{ +public: + sinsp_filter_check_list(); + virtual ~sinsp_filter_check_list(); +}; + +// This is the "default" filter check list +extern sinsp_filter_check_list g_filterlist; + +#endif // HAS_FILTERING diff --git a/userspace/libsinsp/filterchecks.h b/userspace/libsinsp/filterchecks.h index 4e886504f..b001a5d0e 100644 --- a/userspace/libsinsp/filterchecks.h +++ b/userspace/libsinsp/filterchecks.h @@ -198,29 +198,11 @@ class sinsp_filter_check : public gen_event_filter_check std::set s_all_event_types; -friend class sinsp_filter_check_list; +friend class filter_check_list; friend class sinsp_filter_optimizer; friend class chk_compare_helper; }; -// -// Global class that stores the list of filtercheck plugins and offers -// functions to work with it. -// -class sinsp_filter_check_list -{ -public: - sinsp_filter_check_list(); - ~sinsp_filter_check_list(); - void add_filter_check(sinsp_filter_check* filter_check); - void get_all_fields(vector* list); - sinsp_filter_check* new_filter_check_from_another(sinsp_filter_check *chk); - sinsp_filter_check* new_filter_check_from_fldname(const string& name, sinsp* inspector, bool do_exact_check); - -private: - vector m_check_list; -}; - /////////////////////////////////////////////////////////////////////////////// // Filter check classes /////////////////////////////////////////////////////////////////////////////// diff --git a/userspace/libsinsp/plugin.cpp b/userspace/libsinsp/plugin.cpp index a45ccd3ae..37264217a 100755 --- a/userspace/libsinsp/plugin.cpp +++ b/userspace/libsinsp/plugin.cpp @@ -34,8 +34,6 @@ limitations under the License. using namespace std; -extern sinsp_filter_check_list g_filterlist; - /////////////////////////////////////////////////////////////////////////////// // source_plugin filter check implementation // This class implements a dynamic filter check that acts as a bridge to the @@ -351,7 +349,10 @@ std::string sinsp_plugin::version::as_string() const std::to_string(m_version_patch); } -std::shared_ptr sinsp_plugin::register_plugin(sinsp* inspector, string filepath, char* config) +std::shared_ptr sinsp_plugin::register_plugin(sinsp* inspector, + string filepath, + char* config, + filter_check_list &available_checks) { string errstr; std::shared_ptr plugin = create_plugin(filepath, config, errstr); @@ -374,10 +375,10 @@ std::shared_ptr sinsp_plugin::register_plugin(sinsp* inspector, st // Create and register the filter checks associated to this plugin // auto info_filtercheck = new sinsp_filter_check_plugininfo(plugin); - g_filterlist.add_filter_check(info_filtercheck); + available_checks.add_filter_check(info_filtercheck); auto filtercheck = new sinsp_filter_check_plugin(plugin); - g_filterlist.add_filter_check(filtercheck); + available_checks.add_filter_check(filtercheck); return plugin; } diff --git a/userspace/libsinsp/plugin.h b/userspace/libsinsp/plugin.h index 1dc438682..a79932b5f 100755 --- a/userspace/libsinsp/plugin.h +++ b/userspace/libsinsp/plugin.h @@ -26,6 +26,8 @@ limitations under the License. #include +#include "filter_check_list.h" + class sinsp_filter_check_plugin; // Base class for source/extractor plugins. Can not be created directly. @@ -74,8 +76,13 @@ class sinsp_plugin // Create and register a plugin from a shared library pointed // to by filepath, and add it to the inspector. + // Also create filterchecks for fields supported by the plugin + // and add them to the provided filter check list. // The created sinsp_plugin is returned. - static std::shared_ptr register_plugin(sinsp* inspector, std::string filepath, char *config); + static std::shared_ptr register_plugin(sinsp* inspector, + std::string filepath, + char *config, + filter_check_list &available_checks = g_filterlist); // Create a plugin from the dynamic library at the provided // path. On error, the shared_ptr will == NULL and errstr is diff --git a/userspace/libsinsp/sinsp.cpp b/userspace/libsinsp/sinsp.cpp index b6e4fc869..19754e57f 100644 --- a/userspace/libsinsp/sinsp.cpp +++ b/userspace/libsinsp/sinsp.cpp @@ -1820,12 +1820,12 @@ const unordered_map* sinsp::get_grouplist() } #ifdef HAS_FILTERING -void sinsp::get_filtercheck_fields_info(OUT vector* list) +void sinsp::get_filtercheck_fields_info(OUT vector& list) { sinsp_utils::get_filtercheck_fields_info(list); } #else -void sinsp::get_filtercheck_fields_info(OUT vector* list) +void sinsp::get_filtercheck_fields_info(OUT vector& list) { } #endif diff --git a/userspace/libsinsp/sinsp.h b/userspace/libsinsp/sinsp.h index e13363a30..466bd8fe2 100644 --- a/userspace/libsinsp/sinsp.h +++ b/userspace/libsinsp/sinsp.h @@ -455,7 +455,7 @@ class SINSP_PUBLIC sinsp : public capture_stats_source, public wmi_handle_source \brief Populate the given vector with the full list of filter check fields that this version of the library supports. */ - static void get_filtercheck_fields_info(std::vector* list); + static void get_filtercheck_fields_info(std::vector& list); bool has_metrics(); diff --git a/userspace/libsinsp/table.cpp b/userspace/libsinsp/table.cpp index 20dd22e81..e066ed0fe 100644 --- a/userspace/libsinsp/table.cpp +++ b/userspace/libsinsp/table.cpp @@ -21,10 +21,10 @@ limitations under the License. #include "sinsp_int.h" #include "../../driver/ppm_ringbuffer.h" #include "filter.h" +#include "filter_check_list.h" #include "filterchecks.h" #include "table.h" -extern sinsp_filter_check_list g_filterlist; extern sinsp_evttables g_infotables; // diff --git a/userspace/libsinsp/utils.cpp b/userspace/libsinsp/utils.cpp index 86d362c79..50ce73e66 100644 --- a/userspace/libsinsp/utils.cpp +++ b/userspace/libsinsp/utils.cpp @@ -45,6 +45,7 @@ limitations under the License. #include "sinsp_errno.h" #include "sinsp_signal.h" #include "filter.h" +#include "filter_check_list.h" #include "filterchecks.h" #include "protodecoder.h" #include "uri.h" @@ -843,7 +844,7 @@ const struct ppm_param_info* sinsp_utils::find_longest_matching_evt_param(string } #ifdef HAS_FILTERING -void sinsp_utils::get_filtercheck_fields_info(OUT vector* list) +void sinsp_utils::get_filtercheck_fields_info(OUT vector& list) { g_filterlist.get_all_fields(list); } diff --git a/userspace/libsinsp/utils.h b/userspace/libsinsp/utils.h index 7219985ce..a615bdcc9 100644 --- a/userspace/libsinsp/utils.h +++ b/userspace/libsinsp/utils.h @@ -97,7 +97,7 @@ class sinsp_utils // // Get the list of filtercheck fields // - static void get_filtercheck_fields_info(std::vector* list); + static void get_filtercheck_fields_info(std::vector& list); static uint64_t get_current_time_ns(); From ca51f17c5f7ad3ae6d85f10598976eab16bdc223 Mon Sep 17 00:00:00 2001 From: Mark Stemm Date: Fri, 15 Oct 2021 06:45:59 -0700 Subject: [PATCH 035/148] Finish moving token bucket impl from falco to libs It took a while, but we remembered to finish moving the token_bucket from falco engine to libs. There were 2 copies for a while. This brings over one change from the falco copy--to have an optional timer function. Signed-off-by: Mark Stemm --- userspace/libsinsp/token_bucket.cpp | 26 ++++++++++++-------------- userspace/libsinsp/token_bucket.h | 4 +++- 2 files changed, 15 insertions(+), 15 deletions(-) diff --git a/userspace/libsinsp/token_bucket.cpp b/userspace/libsinsp/token_bucket.cpp index 951d53d6b..b999df8f9 100644 --- a/userspace/libsinsp/token_bucket.cpp +++ b/userspace/libsinsp/token_bucket.cpp @@ -12,17 +12,23 @@ distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. - */ #include +#include +#include -#include "sinsp.h" -#include "utils.h" #include "token_bucket.h" +#include "utils.h" + +token_bucket::token_bucket(): + token_bucket(sinsp_utils::get_current_time_ns) +{ +} -token_bucket::token_bucket() +token_bucket::token_bucket(std::function timer) { + m_timer = timer; init(1, 1); } @@ -35,20 +41,12 @@ void token_bucket::init(double rate, double max_tokens, uint64_t now) m_rate = rate; m_max_tokens = max_tokens; m_tokens = max_tokens; - - if(now == 0) - { - now = sinsp_utils::get_current_time_ns(); - } - - m_last_seen = now; + m_last_seen = now == 0 ? m_timer() : now; } bool token_bucket::claim() { - uint64_t now = sinsp_utils::get_current_time_ns(); - - return claim(1, now); + return claim(1, m_timer()); } bool token_bucket::claim(double tokens, uint64_t now) diff --git a/userspace/libsinsp/token_bucket.h b/userspace/libsinsp/token_bucket.h index 8e4e67ce6..132a3592e 100644 --- a/userspace/libsinsp/token_bucket.h +++ b/userspace/libsinsp/token_bucket.h @@ -18,14 +18,15 @@ limitations under the License. #pragma once #include +#include // A simple token bucket that accumulates tokens at a fixed rate and allows // for limited bursting in the form of "banked" tokens. - class token_bucket { public: token_bucket(); + token_bucket(std::function timer); virtual ~token_bucket(); // @@ -51,6 +52,7 @@ class token_bucket uint64_t get_last_seen(); private: + std::function m_timer; // // The number of tokens generated per second. From ca92a8665ae86487e5c89b8e29723c6cdbb4d765 Mon Sep 17 00:00:00 2001 From: Mark Stemm Date: Fri, 15 Oct 2021 12:24:32 -0700 Subject: [PATCH 036/148] Split sinsp_evt_filter_check_event in two, use half in plugins Split sinsp_evt_filter_check_event in two. (New) sinsp_evt_filter_check_gen_event has fields for event number and time. sinsp_evt_filter_check_event has fields for everything else. Add sinsp_evt_filter_check_gen_event to the filtercheck list for plugins so filters/conditions can have fields that work on time/event number. Signed-off-by: Mark Stemm --- userspace/libsinsp/filterchecks.cpp | 157 ++++++++++++++++++---------- userspace/libsinsp/filterchecks.h | 142 ++++++++++++++----------- userspace/libsinsp/plugin.cpp | 3 + 3 files changed, 187 insertions(+), 115 deletions(-) diff --git a/userspace/libsinsp/filterchecks.cpp b/userspace/libsinsp/filterchecks.cpp index 89c0248d7..ccf476f25 100644 --- a/userspace/libsinsp/filterchecks.cpp +++ b/userspace/libsinsp/filterchecks.cpp @@ -2786,7 +2786,7 @@ bool sinsp_filter_check_thread::compare(sinsp_evt *evt) /////////////////////////////////////////////////////////////////////////////// // sinsp_filter_check_event implementation /////////////////////////////////////////////////////////////////////////////// -const filtercheck_field_info sinsp_filter_check_event_fields[] = +const filtercheck_field_info sinsp_filter_check_gen_event_fields[] = { {PT_UINT64, EPF_NONE, PF_ID, "evt.num", "event number."}, {PT_CHARBUF, EPF_NONE, PF_NA, "evt.time", "event timestamp as a time string that includes the nanosecond part."}, @@ -2800,6 +2800,108 @@ const filtercheck_field_info sinsp_filter_check_event_fields[] = {PT_RELTIME, EPF_NONE, PF_10_PADDED_DEC, "evt.reltime", "number of nanoseconds from the beginning of the capture."}, {PT_RELTIME, EPF_NONE, PF_DEC, "evt.reltime.s", "number of seconds from the beginning of the capture."}, {PT_RELTIME, EPF_NONE, PF_10_PADDED_DEC, "evt.reltime.ns", "fractional part (in ns) of the time from the beginning of the capture."}, +}; + +sinsp_filter_check_gen_event::sinsp_filter_check_gen_event() +{ + m_info.m_name = "evt"; + m_info.m_fields = sinsp_filter_check_gen_event_fields; + m_info.m_nfields = sizeof(sinsp_filter_check_gen_event_fields) / sizeof(sinsp_filter_check_gen_event_fields[0]); + m_u64val = 0; +} + +sinsp_filter_check_gen_event::~sinsp_filter_check_gen_event() +{ +} + +sinsp_filter_check* sinsp_filter_check_gen_event::allocate_new() +{ + return (sinsp_filter_check*) new sinsp_filter_check_gen_event(); +} + +Json::Value sinsp_filter_check_gen_event::extract_as_js(sinsp_evt *evt, OUT uint32_t* len) +{ + switch(m_field_id) + { + case TYPE_TIME: + case TYPE_TIME_S: + case TYPE_TIME_ISO8601: + case TYPE_DATETIME: + case TYPE_DATETIME_S: + return (Json::Value::Int64)evt->get_ts(); + + case TYPE_RAWTS: + case TYPE_RAWTS_S: + case TYPE_RAWTS_NS: + case TYPE_RELTS: + case TYPE_RELTS_S: + case TYPE_RELTS_NS: + return (Json::Value::Int64)*(uint64_t*)extract(evt, len); + default: + return Json::nullValue; + } + + return Json::nullValue; +} + +uint8_t* sinsp_filter_check_gen_event::extract(sinsp_evt *evt, OUT uint32_t* len, bool sanitize_strings) +{ + *len = 0; + switch(m_field_id) + { + case TYPE_TIME: + if(false) + { + m_strstorage = to_string(evt->get_ts()); + } + else + { + sinsp_utils::ts_to_string(evt->get_ts(), &m_strstorage, false, true); + } + RETURN_EXTRACT_STRING(m_strstorage); + case TYPE_TIME_S: + sinsp_utils::ts_to_string(evt->get_ts(), &m_strstorage, false, false); + RETURN_EXTRACT_STRING(m_strstorage); + case TYPE_TIME_ISO8601: + sinsp_utils::ts_to_iso_8601(evt->get_ts(), &m_strstorage); + RETURN_EXTRACT_STRING(m_strstorage); + case TYPE_DATETIME: + sinsp_utils::ts_to_string(evt->get_ts(), &m_strstorage, true, true); + RETURN_EXTRACT_STRING(m_strstorage); + case TYPE_DATETIME_S: + sinsp_utils::ts_to_string(evt->get_ts(), &m_strstorage, true, false); + RETURN_EXTRACT_STRING(m_strstorage); + case TYPE_RAWTS: + m_u64val = evt->get_ts(); + RETURN_EXTRACT_VAR(m_u64val); + case TYPE_RAWTS_S: + m_u64val = evt->get_ts() / ONE_SECOND_IN_NS; + RETURN_EXTRACT_VAR(m_u64val); + case TYPE_RAWTS_NS: + m_u64val = evt->get_ts() % ONE_SECOND_IN_NS; + RETURN_EXTRACT_VAR(m_u64val); + case TYPE_RELTS: + m_u64val = evt->get_ts() - m_inspector->m_firstevent_ts; + RETURN_EXTRACT_VAR(m_u64val); + case TYPE_RELTS_S: + m_u64val = (evt->get_ts() - m_inspector->m_firstevent_ts) / ONE_SECOND_IN_NS; + RETURN_EXTRACT_VAR(m_u64val); + case TYPE_RELTS_NS: + m_u64val = (evt->get_ts() - m_inspector->m_firstevent_ts) % ONE_SECOND_IN_NS; + RETURN_EXTRACT_VAR(m_u64val); + case TYPE_NUMBER: + m_u64val = evt->get_num(); + RETURN_EXTRACT_VAR(m_u64val); + default: + ASSERT(false); + return NULL; + } + + return NULL; +} + +const filtercheck_field_info sinsp_filter_check_event_fields[] = +{ {PT_RELTIME, EPF_NONE, PF_DEC, "evt.latency", "delta between an exit event and the correspondent enter event, in nanoseconds."}, {PT_RELTIME, EPF_NONE, PF_DEC, "evt.latency.s", "integer part of the event latency delta."}, {PT_RELTIME, EPF_NONE, PF_10_PADDED_DEC, "evt.latency.ns", "fractional part of the event latency delta."}, @@ -3340,20 +3442,9 @@ Json::Value sinsp_filter_check_event::extract_as_js(sinsp_evt *evt, OUT uint32_t { switch(m_field_id) { - case TYPE_TIME: - case TYPE_TIME_S: - case TYPE_TIME_ISO8601: - case TYPE_DATETIME: - case TYPE_DATETIME_S: case TYPE_RUNTIME_TIME_OUTPUT_FORMAT: return (Json::Value::Int64)evt->get_ts(); - case TYPE_RAWTS: - case TYPE_RAWTS_S: - case TYPE_RAWTS_NS: - case TYPE_RELTS: - case TYPE_RELTS_S: - case TYPE_RELTS_NS: case TYPE_LATENCY: case TYPE_LATENCY_S: case TYPE_LATENCY_NS: @@ -3417,46 +3508,6 @@ uint8_t* sinsp_filter_check_event::extract(sinsp_evt *evt, OUT uint32_t* len, bo *len = 0; switch(m_field_id) { - case TYPE_TIME: -// if(g_filterchecks_force_raw_times) - if(false) - { - m_strstorage = to_string(evt->get_ts()); - } - else - { - sinsp_utils::ts_to_string(evt->get_ts(), &m_strstorage, false, true); - } - RETURN_EXTRACT_STRING(m_strstorage); - case TYPE_TIME_S: - sinsp_utils::ts_to_string(evt->get_ts(), &m_strstorage, false, false); - RETURN_EXTRACT_STRING(m_strstorage); - case TYPE_TIME_ISO8601: - sinsp_utils::ts_to_iso_8601(evt->get_ts(), &m_strstorage); - RETURN_EXTRACT_STRING(m_strstorage); - case TYPE_DATETIME: - sinsp_utils::ts_to_string(evt->get_ts(), &m_strstorage, true, true); - RETURN_EXTRACT_STRING(m_strstorage); - case TYPE_DATETIME_S: - sinsp_utils::ts_to_string(evt->get_ts(), &m_strstorage, true, false); - RETURN_EXTRACT_STRING(m_strstorage); - case TYPE_RAWTS: - RETURN_EXTRACT_VAR(evt->m_pevt->ts); - case TYPE_RAWTS_S: - m_u64val = evt->get_ts() / ONE_SECOND_IN_NS; - RETURN_EXTRACT_VAR(m_u64val); - case TYPE_RAWTS_NS: - m_u64val = evt->get_ts() % ONE_SECOND_IN_NS; - RETURN_EXTRACT_VAR(m_u64val); - case TYPE_RELTS: - m_u64val = evt->get_ts() - m_inspector->m_firstevent_ts; - RETURN_EXTRACT_VAR(m_u64val); - case TYPE_RELTS_S: - m_u64val = (evt->get_ts() - m_inspector->m_firstevent_ts) / ONE_SECOND_IN_NS; - RETURN_EXTRACT_VAR(m_u64val); - case TYPE_RELTS_NS: - m_u64val = (evt->get_ts() - m_inspector->m_firstevent_ts) % ONE_SECOND_IN_NS; - RETURN_EXTRACT_VAR(m_u64val); case TYPE_LATENCY: { m_u64val = 0; @@ -3802,8 +3853,6 @@ uint8_t* sinsp_filter_check_event::extract(sinsp_evt *evt, OUT uint32_t* len, bo } RETURN_EXTRACT_STRING(m_strstorage); - case TYPE_NUMBER: - RETURN_EXTRACT_VAR(evt->m_evtnum); case TYPE_CPU: RETURN_EXTRACT_VAR(evt->m_cpuid); case TYPE_ARGRAW: diff --git a/userspace/libsinsp/filterchecks.h b/userspace/libsinsp/filterchecks.h index b001a5d0e..1e3d72e8d 100644 --- a/userspace/libsinsp/filterchecks.h +++ b/userspace/libsinsp/filterchecks.h @@ -381,9 +381,9 @@ class sinsp_filter_check_thread : public sinsp_filter_check }; // -// event checks +// filterchecks that will work on any generic event // -class sinsp_filter_check_event : public sinsp_filter_check +class sinsp_filter_check_gen_event : public sinsp_filter_check { public: enum check_type @@ -400,65 +400,85 @@ class sinsp_filter_check_event : public sinsp_filter_check TYPE_RELTS = 9, TYPE_RELTS_S = 10, TYPE_RELTS_NS = 11, - TYPE_LATENCY = 12, - TYPE_LATENCY_S = 13, - TYPE_LATENCY_NS = 14, - TYPE_LATENCY_QUANTIZED = 15, - TYPE_LATENCY_HUMAN = 16, - TYPE_DELTA = 17, - TYPE_DELTA_S = 18, - TYPE_DELTA_NS = 19, - TYPE_RUNTIME_TIME_OUTPUT_FORMAT = 20, - TYPE_DIR = 21, - TYPE_TYPE = 22, - TYPE_TYPE_IS = 23, - TYPE_SYSCALL_TYPE = 24, - TYPE_CATEGORY = 25, - TYPE_CPU = 26, - TYPE_ARGS = 27, - TYPE_ARGSTR = 28, - TYPE_ARGRAW = 29, - TYPE_INFO = 30, - TYPE_BUFFER = 31, - TYPE_BUFLEN = 32, - TYPE_RESSTR = 33, - TYPE_RESRAW = 34, - TYPE_FAILED = 35, - TYPE_ISIO = 36, - TYPE_ISIO_READ = 37, - TYPE_ISIO_WRITE = 38, - TYPE_IODIR = 39, - TYPE_ISWAIT = 40, - TYPE_WAIT_LATENCY = 41, - TYPE_ISSYSLOG = 42, - TYPE_COUNT = 43, - TYPE_COUNT_ERROR = 44, - TYPE_COUNT_ERROR_FILE = 45, - TYPE_COUNT_ERROR_NET = 46, - TYPE_COUNT_ERROR_MEMORY = 47, - TYPE_COUNT_ERROR_OTHER = 48, - TYPE_COUNT_EXIT = 49, - TYPE_COUNT_PROCINFO = 50, - TYPE_COUNT_THREADINFO = 51, - TYPE_AROUND = 52, - TYPE_ABSPATH = 53, - TYPE_BUFLEN_IN = 54, - TYPE_BUFLEN_OUT = 55, - TYPE_BUFLEN_FILE = 56, - TYPE_BUFLEN_FILE_IN = 57, - TYPE_BUFLEN_FILE_OUT = 58, - TYPE_BUFLEN_NET = 59, - TYPE_BUFLEN_NET_IN = 60, - TYPE_BUFLEN_NET_OUT = 61, - TYPE_ISOPEN_READ = 62, - TYPE_ISOPEN_WRITE = 63, - TYPE_INFRA_DOCKER_NAME = 64, - TYPE_INFRA_DOCKER_CONTAINER_ID = 65, - TYPE_INFRA_DOCKER_CONTAINER_NAME = 66, - TYPE_INFRA_DOCKER_CONTAINER_IMAGE = 67, - TYPE_ISOPEN_EXEC = 68, - TYPE_PLUGIN_NAME = 69, - TYPE_PLUGIN_INFO = 70, + }; + + sinsp_filter_check_gen_event(); + ~sinsp_filter_check_gen_event(); + sinsp_filter_check* allocate_new(); + uint8_t* extract(sinsp_evt *evt, OUT uint32_t* len, bool sanitize_strings = true); + Json::Value extract_as_js(sinsp_evt *evt, OUT uint32_t* len); + + uint64_t m_u64val; + string m_strstorage; +}; + +// +// event checks +// +class sinsp_filter_check_event : public sinsp_filter_check +{ +public: + enum check_type + { + TYPE_LATENCY = 0, + TYPE_LATENCY_S = 1, + TYPE_LATENCY_NS = 2, + TYPE_LATENCY_QUANTIZED = 3, + TYPE_LATENCY_HUMAN = 4, + TYPE_DELTA = 5, + TYPE_DELTA_S = 6, + TYPE_DELTA_NS = 7, + TYPE_RUNTIME_TIME_OUTPUT_FORMAT = 8, + TYPE_DIR = 9, + TYPE_TYPE = 10, + TYPE_TYPE_IS = 11, + TYPE_SYSCALL_TYPE = 12, + TYPE_CATEGORY = 13, + TYPE_CPU = 14, + TYPE_ARGS = 15, + TYPE_ARGSTR = 16, + TYPE_ARGRAW = 17, + TYPE_INFO = 18, + TYPE_BUFFER = 19, + TYPE_BUFLEN = 20, + TYPE_RESSTR = 21, + TYPE_RESRAW = 22, + TYPE_FAILED = 23, + TYPE_ISIO = 24, + TYPE_ISIO_READ = 25, + TYPE_ISIO_WRITE = 26, + TYPE_IODIR = 27, + TYPE_ISWAIT = 28, + TYPE_WAIT_LATENCY = 29, + TYPE_ISSYSLOG = 30, + TYPE_COUNT = 31, + TYPE_COUNT_ERROR = 32, + TYPE_COUNT_ERROR_FILE = 33, + TYPE_COUNT_ERROR_NET = 34, + TYPE_COUNT_ERROR_MEMORY = 35, + TYPE_COUNT_ERROR_OTHER = 36, + TYPE_COUNT_EXIT = 37, + TYPE_COUNT_PROCINFO = 38, + TYPE_COUNT_THREADINFO = 39, + TYPE_AROUND = 40, + TYPE_ABSPATH = 41, + TYPE_BUFLEN_IN = 42, + TYPE_BUFLEN_OUT = 43, + TYPE_BUFLEN_FILE = 44, + TYPE_BUFLEN_FILE_IN = 45, + TYPE_BUFLEN_FILE_OUT = 46, + TYPE_BUFLEN_NET = 47, + TYPE_BUFLEN_NET_IN = 48, + TYPE_BUFLEN_NET_OUT = 49, + TYPE_ISOPEN_READ = 50, + TYPE_ISOPEN_WRITE = 51, + TYPE_INFRA_DOCKER_NAME = 52, + TYPE_INFRA_DOCKER_CONTAINER_ID = 53, + TYPE_INFRA_DOCKER_CONTAINER_NAME = 54, + TYPE_INFRA_DOCKER_CONTAINER_IMAGE = 55, + TYPE_ISOPEN_EXEC = 56, + TYPE_PLUGIN_NAME = 57, + TYPE_PLUGIN_INFO = 58, }; sinsp_filter_check_event(); diff --git a/userspace/libsinsp/plugin.cpp b/userspace/libsinsp/plugin.cpp index 37264217a..e2b492df5 100755 --- a/userspace/libsinsp/plugin.cpp +++ b/userspace/libsinsp/plugin.cpp @@ -374,6 +374,9 @@ std::shared_ptr sinsp_plugin::register_plugin(sinsp* inspector, // // Create and register the filter checks associated to this plugin // + auto evt_filtercheck = new sinsp_filter_check_gen_event(); + available_checks.add_filter_check(evt_filtercheck); + auto info_filtercheck = new sinsp_filter_check_plugininfo(plugin); available_checks.add_filter_check(info_filtercheck); From a9dd4cf62c473489d3c9ea54d36c107fe362161e Mon Sep 17 00:00:00 2001 From: Mark Stemm Date: Fri, 15 Oct 2021 15:03:00 -0700 Subject: [PATCH 037/148] Divorce plugin rc from scap rc Completely divorce plugin rc values from scap rc values. Instead of being int32_t, plugin rc values are a ss_plugin_rc enum and don't directly relate to SCAP_XXX values. In a couple of locations in scap.c, plugin rcs do need to be mapped to scap rcs to reflect opening sources, getting next event. Create a function plugin_rc_to_scap_rc for that. Also cleaned up a few other cases where ints were being used instead of enums (plugin type, return value from next, etc.) Signed-off-by: Mark Stemm --- userspace/libscap/plugin_info.h | 44 ++++++++++++++----------------- userspace/libscap/scap-int.h | 3 ++- userspace/libscap/scap.c | 46 +++++++++++++++++++++++++++------ userspace/libsinsp/plugin.cpp | 16 ++++++------ userspace/libsinsp/plugin.h | 6 ++--- 5 files changed, 70 insertions(+), 45 deletions(-) diff --git a/userspace/libscap/plugin_info.h b/userspace/libscap/plugin_info.h index c26d8c3e7..f991abf4e 100644 --- a/userspace/libscap/plugin_info.h +++ b/userspace/libscap/plugin_info.h @@ -60,20 +60,15 @@ typedef enum ss_plugin_field_type }ss_plugin_field_type; // Values to return from init() / open() / next() / next_batch() / -// extract_fields(). Note that these values map exactly to the -// corresponding SCAP_XXX values in scap.h, and should be kept in -// sync. -#define SS_PLUGIN_SUCCESS 0 -#define SS_PLUGIN_FAILURE 1 -#define SS_PLUGIN_TIMEOUT -1 -#define SS_PLUGIN_ILLEGAL_INPUT 3 -#define SS_PLUGIN_NOTFOUND 4 -#define SS_PLUGIN_INPUT_TOO_SMALL 5 -#define SS_PLUGIN_EOF 6 -#define SS_PLUGIN_UNEXPECTED_BLOCK 7 -#define SS_PLUGIN_VERSION_MISMATCH 8 -#define SS_PLUGIN_NOT_SUPPORTED 9 - +// extract_fields(). +typedef enum ss_plugin_rc +{ + SS_PLUGIN_SUCCESS = 0, + SS_PLUGIN_FAILURE = 1, + SS_PLUGIN_TIMEOUT = -1, + SS_PLUGIN_EOF = 2, + SS_PLUGIN_NOT_SUPPORTED = 3, +} ss_plugin_rc; // This struct represents an event returned by the plugin, and is used // below in next()/next_batch(). @@ -202,13 +197,13 @@ typedef struct // Arguments: // - config: a string with the plugin configuration. The format of the // string is chosen by the plugin itself. - // - rc: pointer to an integer that will contain the initialization result, + // - rc: pointer to a ss_plugin_rc that will contain the initialization result, // as a SS_PLUGIN_* value (e.g. SS_PLUGIN_SUCCESS=0, SS_PLUGIN_FAILURE=1) // Return value: pointer to the plugin state that will be treated as opaque // by the engine and passed to the other plugin functions. // If rc is SS_PLUGIN_FAILURE, this function should return NULL. // - ss_plugin_t* (*init)(char* config, int32_t* rc); + ss_plugin_t* (*init)(char* config, ss_plugin_rc* rc); // // Destroy the plugin and, if plugin state was allocated, free it. // Required: yes @@ -298,12 +293,12 @@ typedef struct // - s: the plugin state returned by init() // - params: the open parameters, as a string. The format is defined by the plugin // itsef - // - rc: pointer to an integer that will contain the open result, as a SS_PLUGIN_* value - // (e.g. SS_PLUGIN_SUCCESS=0, SS_PLUGIN_FAILURE=1) + // - rc: pointer to a ss_plugin_rc that will contain the open result, + // as a SS_PLUGIN_* value (e.g. SS_PLUGIN_SUCCESS=0, SS_PLUGIN_FAILURE=1) // Return value: a pointer to the open context that will be passed to next(), // close(), event_to_string() and extract_fields. // - ss_instance_t* (*open)(ss_plugin_t* s, char* params, int32_t* rc); + ss_instance_t* (*open)(ss_plugin_t* s, char* params, ss_plugin_rc* rc); // // Close a capture. // Required: yes @@ -328,7 +323,7 @@ typedef struct // Return value: the status of the operation (e.g. SS_PLUGIN_SUCCESS=0, SS_PLUGIN_FAILURE=1, // SS_PLUGIN_TIMEOUT=-1) // - int32_t (*next)(ss_plugin_t* s, ss_instance_t* h, ss_plugin_event **evt); + ss_plugin_rc (*next)(ss_plugin_t* s, ss_instance_t* h, ss_plugin_event **evt); // // Return the read progress. // Required: no @@ -365,9 +360,9 @@ typedef struct // - fields: an array of ss_plugin_extract_field structs. Each element contains // a single field + optional arg, and the corresponding extracted value should // be in the same struct. - // Return value: An integer with values SS_PLUGIN_SUCCESS or SS_PLUGIN_FAILURE. + // Return value: An ss_plugin_rc with values SS_PLUGIN_SUCCESS or SS_PLUGIN_FAILURE. // - int32_t (*extract_fields)(ss_plugin_t *s, const ss_plugin_event *evt, uint32_t num_fields, ss_plugin_extract_field *fields); + ss_plugin_rc (*extract_fields)(ss_plugin_t *s, const ss_plugin_event *evt, uint32_t num_fields, ss_plugin_extract_field *fields); // // This is an optional, internal, function used to speed up event capture by // batching the calls to next(). @@ -380,7 +375,7 @@ typedef struct // owned by the plugin framework and will free them using plugin_free_mem(). // Required: no // - int32_t (*next_batch)(ss_plugin_t* s, ss_instance_t* h, uint32_t *nevts, ss_plugin_event **evts); + ss_plugin_rc (*next_batch)(ss_plugin_t* s, ss_instance_t* h, uint32_t *nevts, ss_plugin_event **evts); // // The following members are PRIVATE for the engine and should not be touched. @@ -499,7 +494,6 @@ typedef struct // array. // char* (*get_fields)(); - // // Extract one or more a filter field values from an event. // Required: no @@ -511,7 +505,7 @@ typedef struct // be in the same struct. // Return value: An integer with values SS_PLUGIN_SUCCESS or SS_PLUGIN_FAILURE. // - int32_t (*extract_fields)(ss_plugin_t *s, const ss_plugin_event *evt, uint32_t num_fields, ss_plugin_extract_field *fields); + ss_plugin_rc (*extract_fields)(ss_plugin_t *s, const ss_plugin_event *evt, uint32_t num_fields, ss_plugin_extract_field *fields); // // The following members are PRIVATE for the engine and should not be touched. diff --git a/userspace/libscap/scap-int.h b/userspace/libscap/scap-int.h index fb712b30a..a5c30681c 100644 --- a/userspace/libscap/scap-int.h +++ b/userspace/libscap/scap-int.h @@ -20,6 +20,7 @@ limitations under the License. //////////////////////////////////////////////////////////////////////////// #include "settings.h" +#include "plugin_info.h" #ifdef __cplusplus extern "C" { @@ -195,7 +196,7 @@ struct scap uint32_t m_input_plugin_batch_idx; // The return value from the last call to batch_next(). - int32_t m_input_plugin_last_batch_res; + ss_plugin_rc m_input_plugin_last_batch_res; }; typedef enum ppm_dumper_type diff --git a/userspace/libscap/scap.c b/userspace/libscap/scap.c index 75ffd44e2..d33dc3c3d 100644 --- a/userspace/libscap/scap.c +++ b/userspace/libscap/scap.c @@ -62,6 +62,34 @@ static const char *SYSDIG_BPF_PROBE_ENV = "SYSDIG_BPF_PROBE"; // #define SCAP_PROBE_VERSION_SIZE 32 +static int32_t plugin_rc_to_scap_rc(ss_plugin_rc plugin_rc) +{ + switch(plugin_rc) + { + case SS_PLUGIN_SUCCESS: + return SCAP_SUCCESS; + break; + case SS_PLUGIN_FAILURE: + return SCAP_FAILURE; + break; + case SS_PLUGIN_TIMEOUT: + return SCAP_TIMEOUT; + break; + case SS_PLUGIN_EOF: + return SCAP_EOF; + break; + case SS_PLUGIN_NOT_SUPPORTED: + return SCAP_NOT_SUPPORTED; + break; + default: + ASSERT(false); + return SCAP_FAILURE; + } + + ASSERT(false); + return SCAP_FAILURE; +} + const char* scap_getlasterr(scap_t* handle) { return handle ? handle->m_lasterr : "null scap handle"; @@ -988,11 +1016,13 @@ scap_t* scap_open_plugin_int(char *error, int32_t *rc, source_plugin_info* input // Set the rc to SCAP_FAILURE now, so in the unlikely event // that a plugin doesn't not actually set a rc, that it gets // treated as a failure. - *rc = SCAP_FAILURE; + ss_plugin_rc plugin_rc = SCAP_FAILURE; handle->m_input_plugin->handle = handle->m_input_plugin->open(handle->m_input_plugin->state, input_plugin_params, - rc); + &plugin_rc); + + *rc = plugin_rc_to_scap_rc(plugin_rc); handle->m_input_plugin_batch_nevts = 0; handle->m_input_plugin_batch_evts = NULL; handle->m_input_plugin_batch_idx = 0; @@ -1717,7 +1747,7 @@ static int32_t scap_next_plugin(scap_t* handle, OUT scap_evt** pevent, OUT uint1 { scap_free_plugin_batch_state(handle); - if(handle->m_input_plugin_last_batch_res != SCAP_SUCCESS) + if(handle->m_input_plugin_last_batch_res != SS_PLUGIN_SUCCESS) { if(handle->m_input_plugin_last_batch_res != SCAP_TIMEOUT && handle->m_input_plugin_last_batch_res != SCAP_EOF) { @@ -1770,17 +1800,17 @@ static int32_t scap_next_plugin(scap_t* handle, OUT scap_evt** pevent, OUT uint1 { should_free_plugin_evt = true; - res = handle->m_input_plugin->next(handle->m_input_plugin->state, - handle->m_input_plugin->handle, &plugin_evt); - if(res != SCAP_SUCCESS) + ss_plugin_rc plugin_res = handle->m_input_plugin->next(handle->m_input_plugin->state, + handle->m_input_plugin->handle, &plugin_evt); + if(plugin_res != SS_PLUGIN_SUCCESS) { - if(res != SCAP_TIMEOUT && res != SCAP_EOF) + if(plugin_res != SS_PLUGIN_TIMEOUT && res != SS_PLUGIN_EOF) { char *errstr = handle->m_input_plugin->get_last_error(handle->m_input_plugin->state); snprintf(handle->m_lasterr, SCAP_LASTERR_SIZE, "%s", errstr); handle->m_input_plugin->free_mem(errstr); } - return res; + return plugin_rc_to_scap_rc(plugin_res); } res = SCAP_SUCCESS; diff --git a/userspace/libsinsp/plugin.cpp b/userspace/libsinsp/plugin.cpp index e2b492df5..874ffd394 100755 --- a/userspace/libsinsp/plugin.cpp +++ b/userspace/libsinsp/plugin.cpp @@ -441,7 +441,7 @@ std::shared_ptr sinsp_plugin::create_plugin(string &filepath, char return ret; } - uint32_t (*get_type)(); + ss_plugin_type (*get_type)(); *(void **) (&get_type) = getsym(handle, "plugin_get_type", errstr); if(get_type == NULL) { @@ -449,7 +449,7 @@ std::shared_ptr sinsp_plugin::create_plugin(string &filepath, char return ret; } - uint32_t plugin_type = get_type(); + ss_plugin_type plugin_type = get_type(); sinsp_source_plugin *splugin; sinsp_extractor_plugin *eplugin; @@ -527,10 +527,10 @@ bool sinsp_plugin::init(char *config) return false; } - int32_t rc; + ss_plugin_rc rc; ss_plugin_t *state = m_plugin_info.init(config, &rc); - if(rc != SCAP_SUCCESS) + if(rc != SS_PLUGIN_SUCCESS) { // Not calling get_last_error here because there was // no valid ss_plugin_t struct returned from init. @@ -615,11 +615,11 @@ bool sinsp_plugin::extract_field(ss_plugin_event &evt, sinsp_plugin::ext_field & efield.arg = field.arg.c_str(); efield.ftype = field.ftype; - int32_t rc; + ss_plugin_rc rc; rc = m_plugin_info.extract_fields(plugin_state(), &evt, num_fields, &efield); - if (rc != SCAP_SUCCESS) + if (rc != SS_PLUGIN_SUCCESS) { return false; } @@ -835,9 +835,9 @@ source_plugin_info *sinsp_source_plugin::plugin_info() return &m_source_plugin_info; } -bool sinsp_source_plugin::open(char *params, int32_t &rc) +bool sinsp_source_plugin::open(char *params, ss_plugin_rc &rc) { - int32_t orc; + ss_plugin_rc orc; if(!plugin_state()) { diff --git a/userspace/libsinsp/plugin.h b/userspace/libsinsp/plugin.h index a79932b5f..626952a22 100755 --- a/userspace/libsinsp/plugin.h +++ b/userspace/libsinsp/plugin.h @@ -136,7 +136,7 @@ class sinsp_plugin typedef struct { char* (*get_required_api_version)(); void (*free_mem)(void *ptr); - ss_plugin_t* (*init)(char* config, int32_t* rc); + ss_plugin_t* (*init)(char* config, ss_plugin_rc* rc); void (*destroy)(ss_plugin_t* s); char* (*get_last_error)(ss_plugin_t* s); char* (*get_name)(); @@ -144,7 +144,7 @@ class sinsp_plugin char* (*get_contact)(); char* (*get_version)(); char* (*get_fields)(); - int32_t (*extract_fields)(ss_plugin_t *s, const ss_plugin_event *evt, uint32_t num_fields, ss_plugin_extract_field *fields); + ss_plugin_rc (*extract_fields)(ss_plugin_t *s, const ss_plugin_event *evt, uint32_t num_fields, ss_plugin_extract_field *fields); } common_plugin_info; std::string m_name; @@ -179,7 +179,7 @@ class sinsp_source_plugin : public sinsp_plugin // Note that embedding ss_instance_t in the object means that // a plugin can only have one open active at a time. - bool open(char *params, int32_t &rc); + bool open(char *params, ss_plugin_rc &rc); void close(); std::string get_progress(uint32_t &progress_pct); From 922a9fe46a118701fa4fe46672ff995911990d69 Mon Sep 17 00:00:00 2001 From: Mark Stemm Date: Wed, 20 Oct 2021 17:24:30 -0700 Subject: [PATCH 038/148] Fix bug in plugin event size as ppm param ppm param sizes are always 16 bits, while the size of a plugin event is 32 bits. So make sure to convert the 32 bit plugin event size to a uint16_t before memcpy()ing. Plugin event lengths are currently guaranteed to be less than 64k by the plugin sdks, although we plan on aligning the sizes (https://github.com/falcosecurity/falco/issues/1759). Also clean up the comments/variable names to improve readability. Signed-off-by: Mark Stemm --- userspace/libscap/scap.c | 18 ++++++++++++------ 1 file changed, 12 insertions(+), 6 deletions(-) diff --git a/userspace/libscap/scap.c b/userspace/libscap/scap.c index d33dc3c3d..6b8653a61 100644 --- a/userspace/libscap/scap.c +++ b/userspace/libscap/scap.c @@ -1816,7 +1816,11 @@ static int32_t scap_next_plugin(scap_t* handle, OUT scap_evt** pevent, OUT uint1 res = SCAP_SUCCESS; } - uint32_t reqsize = sizeof(scap_evt) + 2 + 4 + 2 + plugin_evt->datalen; + // The numbers are: + // - size of plugin id param length (16 bits), holding the value 4 + // - size of event size param (16 bits), holding the event length + // - plugin id (32 bits) + uint32_t reqsize = sizeof(scap_evt) + 2 + 2 + 4 + plugin_evt->datalen; if(handle->m_input_plugin_evt_storage_len < reqsize) { handle->m_input_plugin_evt_storage = (uint8_t*)realloc(handle->m_input_plugin_evt_storage, reqsize); @@ -1831,12 +1835,14 @@ static int32_t scap_next_plugin(scap_t* handle, OUT scap_evt** pevent, OUT uint1 uint8_t* buf = handle->m_input_plugin_evt_storage + sizeof(scap_evt); - const uint16_t four = 4; - memcpy(buf, &four, sizeof(four)); - buf += sizeof(four); + const uint16_t plugin_id_size = 4; + memcpy(buf, &plugin_id_size, sizeof(plugin_id_size)); + buf += sizeof(plugin_id_size); - memcpy(buf, &(plugin_evt->datalen), sizeof(plugin_evt->datalen)); - buf += sizeof(plugin_evt->datalen); + // Plugin event sizes are 32 bits but param sizes are 16 bits + uint16_t datalen = plugin_evt->datalen; + memcpy(buf, &(datalen), sizeof(datalen)); + buf += sizeof(datalen); memcpy(buf, &(handle->m_input_plugin->id), sizeof(handle->m_input_plugin->id)); buf += sizeof(handle->m_input_plugin->id); From b45053da5488c473dc9045db48e478852824acd5 Mon Sep 17 00:00:00 2001 From: Mark Stemm Date: Thu, 21 Oct 2021 10:44:26 -0700 Subject: [PATCH 039/148] Set plugin instance to NULL after close() Make sure to set the plugin instance handle to NULL after calling close(). This ensures that close() will only be called once for a given handle. Signed-off-by: Mark Stemm --- userspace/libscap/scap.c | 1 + userspace/libsinsp/plugin.cpp | 1 + 2 files changed, 2 insertions(+) diff --git a/userspace/libscap/scap.c b/userspace/libscap/scap.c index 6b8653a61..01f48c1d3 100644 --- a/userspace/libscap/scap.c +++ b/userspace/libscap/scap.c @@ -1207,6 +1207,7 @@ void scap_close(scap_t* handle) { handle->m_input_plugin->close(handle->m_input_plugin->state, handle->m_input_plugin->handle); scap_free_plugin_batch_state(handle); + handle->m_input_plugin->state = NULL; // name was allocated handle->m_input_plugin->free_mem(handle->m_input_plugin->name); } diff --git a/userspace/libsinsp/plugin.cpp b/userspace/libsinsp/plugin.cpp index 874ffd394..a7e73a5d4 100755 --- a/userspace/libsinsp/plugin.cpp +++ b/userspace/libsinsp/plugin.cpp @@ -859,6 +859,7 @@ void sinsp_source_plugin::close() } m_source_plugin_info.close(plugin_state(), m_source_plugin_info.handle); + m_source_plugin_info.state = NULL; } std::string sinsp_source_plugin::get_progress(uint32_t &progress_pct) From 8b45f23f4bb9a63dc81f69821198f7f012003421 Mon Sep 17 00:00:00 2001 From: Mark Stemm Date: Thu, 21 Oct 2021 10:45:59 -0700 Subject: [PATCH 040/148] Add missing conversion from plugin rc to scap rc Without this fix, the plugin eof wasn't being interpreted properly as a scap eof. Signed-off-by: Mark Stemm --- userspace/libscap/scap.c | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/userspace/libscap/scap.c b/userspace/libscap/scap.c index 01f48c1d3..813f2cd98 100644 --- a/userspace/libscap/scap.c +++ b/userspace/libscap/scap.c @@ -1761,10 +1761,11 @@ static int32_t scap_next_plugin(scap_t* handle, OUT scap_evt** pevent, OUT uint1 return tres; } - handle->m_input_plugin_last_batch_res = handle->m_input_plugin->next_batch(handle->m_input_plugin->state, - handle->m_input_plugin->handle, - &(handle->m_input_plugin_batch_nevts), - &(handle->m_input_plugin_batch_evts)); + int32_t plugin_res = handle->m_input_plugin->next_batch(handle->m_input_plugin->state, + handle->m_input_plugin->handle, + &(handle->m_input_plugin_batch_nevts), + &(handle->m_input_plugin_batch_evts)); + handle->m_input_plugin_last_batch_res = plugin_rc_to_scap_rc(plugin_res); if(handle->m_input_plugin_batch_nevts == 0) { From 0ccf9cf91085ee8c0e74b7c5a58e2d01e35ef8a4 Mon Sep 17 00:00:00 2001 From: Jason Dellaluce Date: Fri, 22 Oct 2021 14:13:26 +0000 Subject: [PATCH 041/148] fix: correct close cleanup This prevents `destroy` from being called in `libsinsp` due to `state` being set to NULL after closing. Co-authored-by: Leonardo Grasso Signed-off-by: Jason Dellaluce --- userspace/libscap/scap.c | 2 +- userspace/libsinsp/plugin.cpp | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/userspace/libscap/scap.c b/userspace/libscap/scap.c index 813f2cd98..d84d10e82 100644 --- a/userspace/libscap/scap.c +++ b/userspace/libscap/scap.c @@ -1207,7 +1207,7 @@ void scap_close(scap_t* handle) { handle->m_input_plugin->close(handle->m_input_plugin->state, handle->m_input_plugin->handle); scap_free_plugin_batch_state(handle); - handle->m_input_plugin->state = NULL; + handle->m_input_plugin->handle = NULL; // name was allocated handle->m_input_plugin->free_mem(handle->m_input_plugin->name); } diff --git a/userspace/libsinsp/plugin.cpp b/userspace/libsinsp/plugin.cpp index a7e73a5d4..896e6244e 100755 --- a/userspace/libsinsp/plugin.cpp +++ b/userspace/libsinsp/plugin.cpp @@ -859,7 +859,7 @@ void sinsp_source_plugin::close() } m_source_plugin_info.close(plugin_state(), m_source_plugin_info.handle); - m_source_plugin_info.state = NULL; + m_source_plugin_info.handle = NULL; } std::string sinsp_source_plugin::get_progress(uint32_t &progress_pct) From 4ec223c81db1a2733b9938d1be2fcefb2f2ecaba Mon Sep 17 00:00:00 2001 From: lucklypse Date: Tue, 21 Sep 2021 20:01:02 +0000 Subject: [PATCH 042/148] update(libsinsp): update tinydir to 1.2.4 Signed-off-by: lucklypse --- userspace/libsinsp/third-party/tinydir.h | 521 ++++++++++++++++++++--- 1 file changed, 452 insertions(+), 69 deletions(-) diff --git a/userspace/libsinsp/third-party/tinydir.h b/userspace/libsinsp/third-party/tinydir.h index 40f202217..e08eb84ec 100644 --- a/userspace/libsinsp/third-party/tinydir.h +++ b/userspace/libsinsp/third-party/tinydir.h @@ -1,5 +1,9 @@ /* -Copyright (c) 2013-2014, Cong Xu +Copyright (c) 2013-2019, tinydir authors: +- Cong Xu +- Lautis Sun +- Baudouin Feildel +- Andargor All rights reserved. Redistribution and use in source and binary forms, with or without @@ -25,58 +29,187 @@ SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. #ifndef TINYDIR_H #define TINYDIR_H +#ifdef __cplusplus +extern "C" { +#endif + +#if ((defined _UNICODE) && !(defined UNICODE)) +#define UNICODE +#endif + +#if ((defined UNICODE) && !(defined _UNICODE)) +#define _UNICODE +#endif + #include #include #include #ifdef _MSC_VER -#define WIN32_LEAN_AND_MEAN -#include -#pragma warning (disable : 4996) +# ifndef WIN32_LEAN_AND_MEAN +# define WIN32_LEAN_AND_MEAN +# endif +# include +# include +# pragma warning(push) +# pragma warning (disable : 4996) #else -#include -#include +# include +# include +# include +# include +#endif +#ifdef __MINGW32__ +# include #endif /* types */ +/* Windows UNICODE wide character support */ +#if defined _MSC_VER || defined __MINGW32__ +# define _tinydir_char_t TCHAR +# define TINYDIR_STRING(s) _TEXT(s) +# define _tinydir_strlen _tcslen +# define _tinydir_strcpy _tcscpy +# define _tinydir_strcat _tcscat +# define _tinydir_strcmp _tcscmp +# define _tinydir_strrchr _tcsrchr +# define _tinydir_strncmp _tcsncmp +#else +# define _tinydir_char_t char +# define TINYDIR_STRING(s) s +# define _tinydir_strlen strlen +# define _tinydir_strcpy strcpy +# define _tinydir_strcat strcat +# define _tinydir_strcmp strcmp +# define _tinydir_strrchr strrchr +# define _tinydir_strncmp strncmp +#endif + +#if (defined _MSC_VER || defined __MINGW32__) +# include +# define _TINYDIR_PATH_MAX MAX_PATH +#elif defined __linux__ +# include +# ifdef PATH_MAX +# define _TINYDIR_PATH_MAX PATH_MAX +# endif +#elif defined(__unix__) || (defined(__APPLE__) && defined(__MACH__)) +# include +# if defined(BSD) +# include +# ifdef PATH_MAX +# define _TINYDIR_PATH_MAX PATH_MAX +# endif +# endif +#endif + +#ifndef _TINYDIR_PATH_MAX #define _TINYDIR_PATH_MAX 4096 +#endif + #ifdef _MSC_VER /* extra chars for the "\\*" mask */ -#define _TINYDIR_PATH_EXTRA 2 +# define _TINYDIR_PATH_EXTRA 2 #else -#define _TINYDIR_PATH_EXTRA 0 +# define _TINYDIR_PATH_EXTRA 0 #endif + #define _TINYDIR_FILENAME_MAX 256 +#if (defined _MSC_VER || defined __MINGW32__) +#define _TINYDIR_DRIVE_MAX 3 +#endif + #ifdef _MSC_VER -#define strncasecmp _strnicmp +# define _TINYDIR_FUNC static __inline +#elif !defined __STDC_VERSION__ || __STDC_VERSION__ < 199901L +# define _TINYDIR_FUNC static __inline__ #else -#include +# define _TINYDIR_FUNC static inline #endif -#ifdef _MSC_VER -#define _TINYDIR_FUNC static __inline +/* readdir_r usage; define TINYDIR_USE_READDIR_R to use it (if supported) */ +#ifdef TINYDIR_USE_READDIR_R + +/* readdir_r is a POSIX-only function, and may not be available under various + * environments/settings, e.g. MinGW. Use readdir fallback */ +#if _POSIX_C_SOURCE >= 1 || _XOPEN_SOURCE || _BSD_SOURCE || _SVID_SOURCE ||\ + _POSIX_SOURCE +# define _TINYDIR_HAS_READDIR_R +#endif +#if _POSIX_C_SOURCE >= 200112L +# define _TINYDIR_HAS_FPATHCONF +# include +#endif +#if _BSD_SOURCE || _SVID_SOURCE || \ + (_POSIX_C_SOURCE >= 200809L || _XOPEN_SOURCE >= 700) +# define _TINYDIR_HAS_DIRFD +# include +#endif +#if defined _TINYDIR_HAS_FPATHCONF && defined _TINYDIR_HAS_DIRFD &&\ + defined _PC_NAME_MAX +# define _TINYDIR_USE_FPATHCONF +#endif +#if defined __MINGW32__ || !defined _TINYDIR_HAS_READDIR_R ||\ + !(defined _TINYDIR_USE_FPATHCONF || defined NAME_MAX) +# define _TINYDIR_USE_READDIR +#endif + +/* Use readdir by default */ #else -#define _TINYDIR_FUNC static __inline__ +# define _TINYDIR_USE_READDIR +#endif + +/* MINGW32 has two versions of dirent, ASCII and UNICODE*/ +#ifndef _MSC_VER +#if (defined __MINGW32__) && (defined _UNICODE) +#define _TINYDIR_DIR _WDIR +#define _tinydir_dirent _wdirent +#define _tinydir_opendir _wopendir +#define _tinydir_readdir _wreaddir +#define _tinydir_closedir _wclosedir +#else +#define _TINYDIR_DIR DIR +#define _tinydir_dirent dirent +#define _tinydir_opendir opendir +#define _tinydir_readdir readdir +#define _tinydir_closedir closedir +#endif #endif -typedef struct +/* Allow user to use a custom allocator by defining _TINYDIR_MALLOC and _TINYDIR_FREE. */ +#if defined(_TINYDIR_MALLOC) && defined(_TINYDIR_FREE) +#elif !defined(_TINYDIR_MALLOC) && !defined(_TINYDIR_FREE) +#else +#error "Either define both alloc and free or none of them!" +#endif + +#if !defined(_TINYDIR_MALLOC) + #define _TINYDIR_MALLOC(_size) malloc(_size) + #define _TINYDIR_FREE(_ptr) free(_ptr) +#endif /* !defined(_TINYDIR_MALLOC) */ + +typedef struct tinydir_file { - char path[_TINYDIR_PATH_MAX]; - char name[_TINYDIR_FILENAME_MAX]; + _tinydir_char_t path[_TINYDIR_PATH_MAX]; + _tinydir_char_t name[_TINYDIR_FILENAME_MAX]; + _tinydir_char_t *extension; int is_dir; int is_reg; -#ifdef _MSC_VER +#ifndef _MSC_VER +#ifdef __MINGW32__ + struct _stat _s; #else struct stat _s; #endif +#endif } tinydir_file; -typedef struct +typedef struct tinydir_dir { - char path[_TINYDIR_PATH_MAX]; + _tinydir_char_t path[_TINYDIR_PATH_MAX]; int has_next; size_t n_files; @@ -85,8 +218,11 @@ typedef struct HANDLE _h; WIN32_FIND_DATA _f; #else - DIR *_d; - struct dirent *_e; + _TINYDIR_DIR *_d; + struct _tinydir_dirent *_e; +#ifndef _TINYDIR_USE_READDIR + struct _tinydir_dirent *_ep; +#endif #endif } tinydir_dir; @@ -94,9 +230,9 @@ typedef struct /* declarations */ _TINYDIR_FUNC -int tinydir_open(tinydir_dir *dir, const char *path); +int tinydir_open(tinydir_dir *dir, const _tinydir_char_t *path); _TINYDIR_FUNC -int tinydir_open_sorted(tinydir_dir *dir, const char *path); +int tinydir_open_sorted(tinydir_dir *dir, const _tinydir_char_t *path); _TINYDIR_FUNC void tinydir_close(tinydir_dir *dir); @@ -109,21 +245,41 @@ int tinydir_readfile_n(const tinydir_dir *dir, tinydir_file *file, size_t i); _TINYDIR_FUNC int tinydir_open_subdir_n(tinydir_dir *dir, size_t i); +_TINYDIR_FUNC +int tinydir_file_open(tinydir_file *file, const _tinydir_char_t *path); +_TINYDIR_FUNC +void _tinydir_get_ext(tinydir_file *file); _TINYDIR_FUNC int _tinydir_file_cmp(const void *a, const void *b); +#ifndef _MSC_VER +#ifndef _TINYDIR_USE_READDIR +_TINYDIR_FUNC +size_t _tinydir_dirent_buf_size(_TINYDIR_DIR *dirp); +#endif +#endif /* definitions*/ _TINYDIR_FUNC -int tinydir_open(tinydir_dir *dir, const char *path) +int tinydir_open(tinydir_dir *dir, const _tinydir_char_t *path) { - if (dir == NULL || path == NULL || strlen(path) == 0) +#ifndef _MSC_VER +#ifndef _TINYDIR_USE_READDIR + int error; + int size; /* using int size */ +#endif +#else + _tinydir_char_t path_buf[_TINYDIR_PATH_MAX]; +#endif + _tinydir_char_t *pathp; + + if (dir == NULL || path == NULL || _tinydir_strlen(path) == 0) { errno = EINVAL; return -1; } - if (strlen(path) + _TINYDIR_PATH_EXTRA >= _TINYDIR_PATH_MAX) + if (_tinydir_strlen(path) + _TINYDIR_PATH_EXTRA >= _TINYDIR_PATH_MAX) { errno = ENAMETOOLONG; return -1; @@ -135,28 +291,54 @@ int tinydir_open(tinydir_dir *dir, const char *path) dir->_h = INVALID_HANDLE_VALUE; #else dir->_d = NULL; +#ifndef _TINYDIR_USE_READDIR + dir->_ep = NULL; +#endif #endif tinydir_close(dir); - strcpy(dir->path, path); + _tinydir_strcpy(dir->path, path); + /* Remove trailing slashes */ + pathp = &dir->path[_tinydir_strlen(dir->path) - 1]; + while (pathp != dir->path && (*pathp == TINYDIR_STRING('\\') || *pathp == TINYDIR_STRING('/'))) + { + *pathp = TINYDIR_STRING('\0'); + pathp++; + } #ifdef _MSC_VER - strcat(dir->path, "\\*"); - dir->_h = FindFirstFile(dir->path, &dir->_f); - dir->path[strlen(dir->path) - 2] = '\0'; - if (dir->_h == INVALID_HANDLE_VALUE) + _tinydir_strcpy(path_buf, dir->path); + _tinydir_strcat(path_buf, TINYDIR_STRING("\\*")); +#if (defined WINAPI_FAMILY) && (WINAPI_FAMILY != WINAPI_FAMILY_DESKTOP_APP) + dir->_h = FindFirstFileEx(path_buf, FindExInfoStandard, &dir->_f, FindExSearchNameMatch, NULL, 0); #else - dir->_d = opendir(path); - if (dir->_d == NULL) + dir->_h = FindFirstFile(path_buf, &dir->_f); #endif + if (dir->_h == INVALID_HANDLE_VALUE) { errno = ENOENT; +#else + dir->_d = _tinydir_opendir(path); + if (dir->_d == NULL) + { +#endif goto bail; } /* read first file */ dir->has_next = 1; #ifndef _MSC_VER - dir->_e = readdir(dir->_d); +#ifdef _TINYDIR_USE_READDIR + dir->_e = _tinydir_readdir(dir->_d); +#else + /* allocate dirent buffer for readdir_r */ + size = _tinydir_dirent_buf_size(dir->_d); /* conversion to int */ + if (size == -1) return -1; + dir->_ep = (struct _tinydir_dirent*)_TINYDIR_MALLOC(size); + if (dir->_ep == NULL) return -1; + + error = readdir_r(dir->_d, dir->_ep, &dir->_e); + if (error != 0) return -1; +#endif if (dir->_e == NULL) { dir->has_next = 0; @@ -171,7 +353,7 @@ int tinydir_open(tinydir_dir *dir, const char *path) } _TINYDIR_FUNC -int tinydir_open_sorted(tinydir_dir *dir, const char *path) +int tinydir_open_sorted(tinydir_dir *dir, const _tinydir_char_t *path) { /* Count the number of files first, to pre-allocate the files array */ size_t n_files = 0; @@ -189,16 +371,15 @@ int tinydir_open_sorted(tinydir_dir *dir, const char *path) } tinydir_close(dir); - if (tinydir_open(dir, path) == -1) + if (n_files == 0 || tinydir_open(dir, path) == -1) { return -1; } dir->n_files = 0; - dir->_files = (tinydir_file *)malloc(sizeof *dir->_files * n_files); + dir->_files = (tinydir_file *)_TINYDIR_MALLOC(sizeof *dir->_files * n_files); if (dir->_files == NULL) { - errno = ENOMEM; goto bail; } while (dir->has_next) @@ -245,10 +426,7 @@ void tinydir_close(tinydir_dir *dir) memset(dir->path, 0, sizeof(dir->path)); dir->has_next = 0; dir->n_files = 0; - if (dir->_files != NULL) - { - free(dir->_files); - } + _TINYDIR_FREE(dir->_files); dir->_files = NULL; #ifdef _MSC_VER if (dir->_h != INVALID_HANDLE_VALUE) @@ -259,10 +437,14 @@ void tinydir_close(tinydir_dir *dir) #else if (dir->_d) { - closedir(dir->_d); + _tinydir_closedir(dir->_d); } dir->_d = NULL; dir->_e = NULL; +#ifndef _TINYDIR_USE_READDIR + _TINYDIR_FREE(dir->_ep); + dir->_ep = NULL; +#endif #endif } @@ -283,7 +465,18 @@ int tinydir_next(tinydir_dir *dir) #ifdef _MSC_VER if (FindNextFile(dir->_h, &dir->_f) == 0) #else - dir->_e = readdir(dir->_d); +#ifdef _TINYDIR_USE_READDIR + dir->_e = _tinydir_readdir(dir->_d); +#else + if (dir->_ep == NULL) + { + return -1; + } + if (readdir_r(dir->_d, dir->_ep, &dir->_e) != 0) + { + return -1; + } +#endif if (dir->_e == NULL) #endif { @@ -305,6 +498,7 @@ int tinydir_next(tinydir_dir *dir) _TINYDIR_FUNC int tinydir_readfile(const tinydir_dir *dir, tinydir_file *file) { + const _tinydir_char_t *filename; if (dir == NULL || file == NULL) { errno = EINVAL; @@ -319,48 +513,48 @@ int tinydir_readfile(const tinydir_dir *dir, tinydir_file *file) errno = ENOENT; return -1; } - if (strlen(dir->path) + - strlen( + filename = #ifdef _MSC_VER - dir->_f.cFileName + dir->_f.cFileName; #else - dir->_e->d_name + dir->_e->d_name; #endif - ) + 1 + _TINYDIR_PATH_EXTRA >= + if (_tinydir_strlen(dir->path) + + _tinydir_strlen(filename) + 1 + _TINYDIR_PATH_EXTRA >= _TINYDIR_PATH_MAX) { /* the path for the file will be too long */ errno = ENAMETOOLONG; return -1; } - if (strlen( -#ifdef _MSC_VER - dir->_f.cFileName -#else - dir->_e->d_name -#endif - ) >= _TINYDIR_FILENAME_MAX) + if (_tinydir_strlen(filename) >= _TINYDIR_FILENAME_MAX) { errno = ENAMETOOLONG; return -1; } - strcpy(file->path, dir->path); - strcat(file->path, "/"); - strcpy(file->name, -#ifdef _MSC_VER - dir->_f.cFileName + _tinydir_strcpy(file->path, dir->path); + if (_tinydir_strcmp(dir->path, TINYDIR_STRING("/")) != 0) + _tinydir_strcat(file->path, TINYDIR_STRING("/")); + _tinydir_strcpy(file->name, filename); + _tinydir_strcat(file->path, filename); +#ifndef _MSC_VER +#ifdef __MINGW32__ + if (_tstat( +#elif (defined _BSD_SOURCE) || (defined _DEFAULT_SOURCE) \ + || ((defined _XOPEN_SOURCE) && (_XOPEN_SOURCE >= 500)) \ + || ((defined _POSIX_C_SOURCE) && (_POSIX_C_SOURCE >= 200112L)) + if (lstat( #else - dir->_e->d_name + if (stat( #endif - ); - strcat(file->path, file->name); -#ifndef _MSC_VER - if (stat(file->path, &file->_s) == -1) + file->path, &file->_s) == -1) { return -1; } #endif + _tinydir_get_ext(file); + file->is_dir = #ifdef _MSC_VER !!(dir->_f.dwFileAttributes & FILE_ATTRIBUTE_DIRECTORY); @@ -404,6 +598,7 @@ int tinydir_readfile_n(const tinydir_dir *dir, tinydir_file *file, size_t i) } memcpy(file, &dir->_files[i], sizeof(tinydir_file)); + _tinydir_get_ext(file); return 0; } @@ -411,7 +606,7 @@ int tinydir_readfile_n(const tinydir_dir *dir, tinydir_file *file, size_t i) _TINYDIR_FUNC int tinydir_open_subdir_n(tinydir_dir *dir, size_t i) { - char path[_TINYDIR_PATH_MAX]; + _tinydir_char_t path[_TINYDIR_PATH_MAX]; if (dir == NULL) { errno = EINVAL; @@ -423,7 +618,7 @@ int tinydir_open_subdir_n(tinydir_dir *dir, size_t i) return -1; } - strcpy(path, dir->_files[i].path); + _tinydir_strcpy(path, dir->_files[i].path); tinydir_close(dir); if (tinydir_open_sorted(dir, path) == -1) { @@ -433,6 +628,145 @@ int tinydir_open_subdir_n(tinydir_dir *dir, size_t i) return 0; } +/* Open a single file given its path */ +_TINYDIR_FUNC +int tinydir_file_open(tinydir_file *file, const _tinydir_char_t *path) +{ + tinydir_dir dir; + int result = 0; + int found = 0; + _tinydir_char_t dir_name_buf[_TINYDIR_PATH_MAX]; + _tinydir_char_t file_name_buf[_TINYDIR_FILENAME_MAX]; + _tinydir_char_t *dir_name; + _tinydir_char_t *base_name; +#if (defined _MSC_VER || defined __MINGW32__) + _tinydir_char_t drive_buf[_TINYDIR_PATH_MAX]; + _tinydir_char_t ext_buf[_TINYDIR_FILENAME_MAX]; +#endif + + if (file == NULL || path == NULL || _tinydir_strlen(path) == 0) + { + errno = EINVAL; + return -1; + } + if (_tinydir_strlen(path) + _TINYDIR_PATH_EXTRA >= _TINYDIR_PATH_MAX) + { + errno = ENAMETOOLONG; + return -1; + } + + /* Get the parent path */ +#if (defined _MSC_VER || defined __MINGW32__) +#if ((defined _MSC_VER) && (_MSC_VER >= 1400)) + errno = _tsplitpath_s( + path, + drive_buf, _TINYDIR_DRIVE_MAX, + dir_name_buf, _TINYDIR_FILENAME_MAX, + file_name_buf, _TINYDIR_FILENAME_MAX, + ext_buf, _TINYDIR_FILENAME_MAX); +#else + _tsplitpath( + path, + drive_buf, + dir_name_buf, + file_name_buf, + ext_buf); +#endif + + if (errno) + { + return -1; + } + +/* _splitpath_s not work fine with only filename and widechar support */ +#ifdef _UNICODE + if (drive_buf[0] == L'\xFEFE') + drive_buf[0] = '\0'; + if (dir_name_buf[0] == L'\xFEFE') + dir_name_buf[0] = '\0'; +#endif + + /* Emulate the behavior of dirname by returning "." for dir name if it's + empty */ + if (drive_buf[0] == '\0' && dir_name_buf[0] == '\0') + { + _tinydir_strcpy(dir_name_buf, TINYDIR_STRING(".")); + } + /* Concatenate the drive letter and dir name to form full dir name */ + _tinydir_strcat(drive_buf, dir_name_buf); + dir_name = drive_buf; + /* Concatenate the file name and extension to form base name */ + _tinydir_strcat(file_name_buf, ext_buf); + base_name = file_name_buf; +#else + _tinydir_strcpy(dir_name_buf, path); + dir_name = dirname(dir_name_buf); + _tinydir_strcpy(file_name_buf, path); + base_name = basename(file_name_buf); +#endif + + /* Special case: if the path is a root dir, open the parent dir as the file */ +#if (defined _MSC_VER || defined __MINGW32__) + if (_tinydir_strlen(base_name) == 0) +#else + if ((_tinydir_strcmp(base_name, TINYDIR_STRING("/"))) == 0) +#endif + { + memset(file, 0, sizeof * file); + file->is_dir = 1; + file->is_reg = 0; + _tinydir_strcpy(file->path, dir_name); + file->extension = file->path + _tinydir_strlen(file->path); + return 0; + } + + /* Open the parent directory */ + if (tinydir_open(&dir, dir_name) == -1) + { + return -1; + } + + /* Read through the parent directory and look for the file */ + while (dir.has_next) + { + if (tinydir_readfile(&dir, file) == -1) + { + result = -1; + goto bail; + } + if (_tinydir_strcmp(file->name, base_name) == 0) + { + /* File found */ + found = 1; + break; + } + tinydir_next(&dir); + } + if (!found) + { + result = -1; + errno = ENOENT; + } + +bail: + tinydir_close(&dir); + return result; +} + +_TINYDIR_FUNC +void _tinydir_get_ext(tinydir_file *file) +{ + _tinydir_char_t *period = _tinydir_strrchr(file->name, TINYDIR_STRING('.')); + if (period == NULL) + { + file->extension = &(file->name[_tinydir_strlen(file->name)]); + } + else + { + file->extension = period + 1; + } +} + _TINYDIR_FUNC int _tinydir_file_cmp(const void *a, const void *b) { @@ -442,7 +776,56 @@ int _tinydir_file_cmp(const void *a, const void *b) { return -(fa->is_dir - fb->is_dir); } - return strncasecmp(fa->name, fb->name, _TINYDIR_FILENAME_MAX); + return _tinydir_strncmp(fa->name, fb->name, _TINYDIR_FILENAME_MAX); } +#ifndef _MSC_VER +#ifndef _TINYDIR_USE_READDIR +/* +The following authored by Ben Hutchings +from https://womble.decadent.org.uk/readdir_r-advisory.html +*/ +/* Calculate the required buffer size (in bytes) for directory * +* entries read from the given directory handle. Return -1 if this * +* this cannot be done. * +* * +* This code does not trust values of NAME_MAX that are less than * +* 255, since some systems (including at least HP-UX) incorrectly * +* define it to be a smaller value. */ +_TINYDIR_FUNC +size_t _tinydir_dirent_buf_size(_TINYDIR_DIR *dirp) +{ + long name_max; + size_t name_end; + /* parameter may be unused */ + (void)dirp; + +#if defined _TINYDIR_USE_FPATHCONF + name_max = fpathconf(dirfd(dirp), _PC_NAME_MAX); + if (name_max == -1) +#if defined(NAME_MAX) + name_max = (NAME_MAX > 255) ? NAME_MAX : 255; +#else + return (size_t)(-1); +#endif +#elif defined(NAME_MAX) + name_max = (NAME_MAX > 255) ? NAME_MAX : 255; +#else +#error "buffer size for readdir_r cannot be determined" +#endif + name_end = (size_t)offsetof(struct _tinydir_dirent, d_name) + name_max + 1; + return (name_end > sizeof(struct _tinydir_dirent) ? + name_end : sizeof(struct _tinydir_dirent)); +} +#endif +#endif + +#ifdef __cplusplus +} +#endif + +# if defined (_MSC_VER) +# pragma warning(pop) +# endif + #endif From 4e76d3f3830e9a028a090f64cc3525df7eeb224d Mon Sep 17 00:00:00 2001 From: Luca Guerra Date: Fri, 22 Oct 2021 11:06:21 +0000 Subject: [PATCH 043/148] userspace: add strlcpy implementation Signed-off-by: Luca Guerra --- userspace/common/strlcpy.h | 42 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 42 insertions(+) create mode 100644 userspace/common/strlcpy.h diff --git a/userspace/common/strlcpy.h b/userspace/common/strlcpy.h new file mode 100644 index 000000000..7a9f44360 --- /dev/null +++ b/userspace/common/strlcpy.h @@ -0,0 +1,42 @@ +/* +Copyright (C) 2021 The Falco Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +#include +#include + + +/*! + \brief Copy up to size - 1 characters from the NUL-terminated string src to dst, NUL-terminating the result. + + \return The length of the source string. +*/ +static inline size_t strlcpy(char *dst, const char *src, size_t size) { + size_t srcsize = strlen(src); + if (size == 0) { + return srcsize; + } + + size_t copysize = srcsize; + + if (copysize > size - 1) { + copysize = size - 1; + } + + memcpy(dst, src, copysize); + dst[copysize] = '\0'; + + return srcsize; +} From 21588dd885a2503ae93739dcc15e83fca6496fa2 Mon Sep 17 00:00:00 2001 From: Luca Guerra Date: Thu, 21 Oct 2021 13:51:42 +0000 Subject: [PATCH 044/148] libsinsp, libscap: use common strlcpy implementation Signed-off-by: Luca Guerra --- userspace/chisel/chisel_api.cpp | 2 +- userspace/libscap/scap.c | 3 ++- userspace/libscap/scap_fds.c | 11 ++++++----- userspace/libscap/scap_iflist.c | 5 +++-- userspace/libscap/scap_userlist.c | 9 +++++---- userspace/libsinsp/filter.cpp | 2 +- userspace/libsinsp/logger.cpp | 2 +- userspace/libsinsp/plugin.cpp | 4 ++-- userspace/libsinsp/sinsp.cpp | 2 +- userspace/libsinsp/sinsp_curl.cpp | 2 +- userspace/libsinsp/socket_handler.h | 4 ++-- userspace/libsinsp/threadinfo.cpp | 6 +++--- userspace/libsinsp/utils.h | 1 + 13 files changed, 29 insertions(+), 24 deletions(-) diff --git a/userspace/chisel/chisel_api.cpp b/userspace/chisel/chisel_api.cpp index 20f074371..4a965f6f1 100644 --- a/userspace/chisel/chisel_api.cpp +++ b/userspace/chisel/chisel_api.cpp @@ -171,7 +171,7 @@ uint32_t lua_cbacks::rawval_to_lua_stack(lua_State *ls, uint8_t* rawval, ppm_par strcpy(address, ""); } - strncpy(ch->m_lua_fld_storage, + strlcpy(ch->m_lua_fld_storage, address, sizeof(ch->m_lua_fld_storage)); diff --git a/userspace/libscap/scap.c b/userspace/libscap/scap.c index d84d10e82..c3cb7536a 100644 --- a/userspace/libscap/scap.c +++ b/userspace/libscap/scap.c @@ -35,6 +35,7 @@ limitations under the License. #endif // _WIN32 #include "scap.h" +#include "../common/strlcpy.h" #ifdef HAS_CAPTURE #if !defined(_WIN32) && !defined(CYGWING_AGENT) #include "driver_config.h" @@ -2553,7 +2554,7 @@ const char* scap_get_host_root() static char env_str[SCAP_MAX_PATH_SIZE + 1]; static bool inited = false; if (! inited) { - strncpy(env_str, p ? p : "", SCAP_MAX_PATH_SIZE); + strlcpy(env_str, p ? p : "", SCAP_MAX_PATH_SIZE); env_str[SCAP_MAX_PATH_SIZE] = '\0'; inited = true; } diff --git a/userspace/libscap/scap_fds.c b/userspace/libscap/scap_fds.c index db2f8a53d..2358e8855 100644 --- a/userspace/libscap/scap_fds.c +++ b/userspace/libscap/scap_fds.c @@ -20,6 +20,7 @@ limitations under the License. #include "scap.h" #include "scap-int.h" #include "scap_savefile.h" +#include "../common/strlcpy.h" #include #include #include @@ -750,7 +751,7 @@ int32_t scap_fd_handle_pipe(scap_t *handle, char *fname, scap_threadinfo *tinfo, } ino = sb.st_ino; } - strncpy(fdi->info.fname, link_name, SCAP_MAX_PATH_SIZE); + strlcpy(fdi->info.fname, link_name, SCAP_MAX_PATH_SIZE); fdi->ino = ino; return scap_add_fd_to_proc_table(handle, tinfo, fdi, error); @@ -976,11 +977,11 @@ int32_t scap_fd_handle_regular_file(scap_t *handle, char *fname, scap_threadinfo else if(fdi->type == SCAP_FD_FILE_V2) { scap_fd_flags_file(handle, fdi, procdir); - strncpy(fdi->info.regularinfo.fname, link_name, SCAP_MAX_PATH_SIZE); + strlcpy(fdi->info.regularinfo.fname, link_name, SCAP_MAX_PATH_SIZE); } else { - strncpy(fdi->info.fname, link_name, SCAP_MAX_PATH_SIZE); + strlcpy(fdi->info.fname, link_name, SCAP_MAX_PATH_SIZE); } return scap_add_fd_to_proc_table(handle, tinfo, fdi, error); @@ -1034,7 +1035,7 @@ int32_t scap_fd_handle_socket(scap_t *handle, char *fname, scap_threadinfo *tinf link_name[r] = '\0'; - strncpy(fdi->info.fname, link_name, SCAP_MAX_PATH_SIZE); + strlcpy(fdi->info.fname, link_name, SCAP_MAX_PATH_SIZE); // link name for sockets should be of the format socket:[ino] if(1 != sscanf(link_name, "socket:[%"PRIi64"]", &ino)) @@ -1168,7 +1169,7 @@ int32_t scap_fd_read_unix_sockets_from_proc_fs(scap_t *handle, const char* filen token = strtok_r(NULL, delimiters, &scratch); if(NULL != token) { - strncpy(fdinfo->info.unix_socket_info.fname, token, SCAP_MAX_PATH_SIZE); + strlcpy(fdinfo->info.unix_socket_info.fname, token, SCAP_MAX_PATH_SIZE); } else { diff --git a/userspace/libscap/scap_iflist.c b/userspace/libscap/scap_iflist.c index 31e10174d..014c65481 100644 --- a/userspace/libscap/scap_iflist.c +++ b/userspace/libscap/scap_iflist.c @@ -19,6 +19,7 @@ limitations under the License. #include "scap.h" #include "scap-int.h" +#include "../common/strlcpy.h" #if defined(HAS_CAPTURE) && !defined(_WIN32) #include @@ -168,7 +169,7 @@ int32_t scap_create_iflist(scap_t* handle) #else handle->m_addrlist->v4list[ifcnt4].bcast = 0; #endif - strncpy(handle->m_addrlist->v4list[ifcnt4].ifname, tempIfAddr->ifa_name, SCAP_MAX_PATH_SIZE); + strlcpy(handle->m_addrlist->v4list[ifcnt4].ifname, tempIfAddr->ifa_name, SCAP_MAX_PATH_SIZE); handle->m_addrlist->v4list[ifcnt4].ifnamelen = strlen(tempIfAddr->ifa_name); handle->m_addrlist->v4list[ifcnt4].linkspeed = 0; @@ -210,7 +211,7 @@ int32_t scap_create_iflist(scap_t* handle) handle->m_addrlist->v4list[ifcnt4].bcast = 0; #endif - strncpy(handle->m_addrlist->v6list[ifcnt6].ifname, tempIfAddr->ifa_name, SCAP_MAX_PATH_SIZE); + strlcpy(handle->m_addrlist->v6list[ifcnt6].ifname, tempIfAddr->ifa_name, SCAP_MAX_PATH_SIZE); handle->m_addrlist->v6list[ifcnt6].ifnamelen = strlen(tempIfAddr->ifa_name); handle->m_addrlist->v6list[ifcnt6].linkspeed = 0; diff --git a/userspace/libscap/scap_userlist.c b/userspace/libscap/scap_userlist.c index 58e8cb870..5fe3521d6 100644 --- a/userspace/libscap/scap_userlist.c +++ b/userspace/libscap/scap_userlist.c @@ -18,6 +18,7 @@ limitations under the License. #include #include "scap.h" #include "scap-int.h" +#include "../common/strlcpy.h" #if defined(HAS_CAPTURE) && !defined(_WIN32) #include @@ -103,7 +104,7 @@ int32_t scap_create_userlist(scap_t* handle) if(p->pw_name) { - strncpy(handle->m_userlist->users[usercnt].name, p->pw_name, sizeof(handle->m_userlist->users[usercnt].name)); + strlcpy(handle->m_userlist->users[usercnt].name, p->pw_name, sizeof(handle->m_userlist->users[usercnt].name)); } else { @@ -112,7 +113,7 @@ int32_t scap_create_userlist(scap_t* handle) if(p->pw_dir) { - strncpy(handle->m_userlist->users[usercnt].homedir, p->pw_dir, sizeof(handle->m_userlist->users[usercnt].homedir)); + strlcpy(handle->m_userlist->users[usercnt].homedir, p->pw_dir, sizeof(handle->m_userlist->users[usercnt].homedir)); } else { @@ -121,7 +122,7 @@ int32_t scap_create_userlist(scap_t* handle) if(p->pw_shell) { - strncpy(handle->m_userlist->users[usercnt].shell, p->pw_shell, sizeof(handle->m_userlist->users[usercnt].shell)); + strlcpy(handle->m_userlist->users[usercnt].shell, p->pw_shell, sizeof(handle->m_userlist->users[usercnt].shell)); } else { @@ -149,7 +150,7 @@ int32_t scap_create_userlist(scap_t* handle) if(g->gr_name) { - strncpy(handle->m_userlist->groups[grpcnt].name, g->gr_name, sizeof(handle->m_userlist->groups[grpcnt].name)); + strlcpy(handle->m_userlist->groups[grpcnt].name, g->gr_name, sizeof(handle->m_userlist->groups[grpcnt].name)); } else { diff --git a/userspace/libsinsp/filter.cpp b/userspace/libsinsp/filter.cpp index a3ce55816..e3ed1dfaa 100644 --- a/userspace/libsinsp/filter.cpp +++ b/userspace/libsinsp/filter.cpp @@ -1015,7 +1015,7 @@ char* sinsp_filter_check::rawval_to_string(uint8_t* rawval, strcpy(address, ""); } - strncpy(m_getpropertystr_storage, + strlcpy(m_getpropertystr_storage, address, 100); diff --git a/userspace/libsinsp/logger.cpp b/userspace/libsinsp/logger.cpp index 9e7946ca7..80240d344 100644 --- a/userspace/libsinsp/logger.cpp +++ b/userspace/libsinsp/logger.cpp @@ -180,7 +180,7 @@ void sinsp_logger::log(std::string msg, const severity sev) if(m_flags & sinsp_logger::OT_ENCODE_SEV) { char sev_buf[ENCODE_LEN + 1]; - strncpy(sev_buf, encode_severity(sev), sizeof(sev_buf)); + strlcpy(sev_buf, encode_severity(sev), sizeof(sev_buf)); sev_buf[sizeof(sev_buf) - 1] = 0; msg.insert(0, sev_buf); } diff --git a/userspace/libsinsp/plugin.cpp b/userspace/libsinsp/plugin.cpp index 896e6244e..2fb2bd3cb 100755 --- a/userspace/libsinsp/plugin.cpp +++ b/userspace/libsinsp/plugin.cpp @@ -765,8 +765,8 @@ bool sinsp_plugin::resolve_dylib_symbols(void *handle, std::string &errstr) throw sinsp_exception(string("error in plugin ") + m_name + ": field JSON entry has no desc"); } - strncpy(tf.m_name, fname.c_str(), sizeof(tf.m_name)); - strncpy(tf.m_description, fdesc.c_str(), sizeof(tf.m_description)); + strlcpy(tf.m_name, fname.c_str(), sizeof(tf.m_name)); + strlcpy(tf.m_description, fdesc.c_str(), sizeof(tf.m_description)); tf.m_print_format = PF_DEC; if(ftype == "string") { diff --git a/userspace/libsinsp/sinsp.cpp b/userspace/libsinsp/sinsp.cpp index 19754e57f..82347fc6a 100644 --- a/userspace/libsinsp/sinsp.cpp +++ b/userspace/libsinsp/sinsp.cpp @@ -589,7 +589,7 @@ int64_t sinsp::get_file_size(const std::string& fname, char *error) } #endif if(errdesc.empty()) errdesc = get_error_desc(err_str); - strncpy(error, errdesc.c_str(), errdesc.size() > SCAP_LASTERR_SIZE ? SCAP_LASTERR_SIZE : errdesc.size()); + strlcpy(error, errdesc.c_str(), errdesc.size() > SCAP_LASTERR_SIZE ? SCAP_LASTERR_SIZE : errdesc.size()); return -1; } diff --git a/userspace/libsinsp/sinsp_curl.cpp b/userspace/libsinsp/sinsp_curl.cpp index c04dbbf07..7c41a7598 100644 --- a/userspace/libsinsp/sinsp_curl.cpp +++ b/userspace/libsinsp/sinsp_curl.cpp @@ -239,7 +239,7 @@ size_t sinsp_curl::header_callback(char *buffer, size_t size, size_t nitems, voi if(sz < CURL_MAX_HTTP_HEADER) { g_logger.log("HTTP redirect Location: (" + buf + ')', sinsp_logger::SEV_TRACE); - std::strncpy((char*) userdata, buf.data(), sz); + strlcpy((char*) userdata, buf.data(), sz); ((char*) userdata)[sz] = 0; } } diff --git a/userspace/libsinsp/socket_handler.h b/userspace/libsinsp/socket_handler.h index df3ab51be..441649907 100644 --- a/userspace/libsinsp/socket_handler.h +++ b/userspace/libsinsp/socket_handler.h @@ -923,7 +923,7 @@ class socket_data_handler std::memset(buf, 0, size); int pass_len = static_cast(strlen((char*)pass)); if(size < (pass_len) + 1) { return 0; } - strncpy(buf, (const char*)pass, pass_len); + strlcpy(buf, (const char*)pass, pass_len); return pass_len; } return 0; @@ -1382,7 +1382,7 @@ class socket_data_handler throw sinsp_exception("Invalid address (too long): [" + m_url.get_path() + ']'); } m_file_addr.sun_family = AF_UNIX; - strncpy(m_file_addr.sun_path, m_url.get_path().c_str(), m_url.get_path().length()); + strlcpy(m_file_addr.sun_path, m_url.get_path().c_str(), m_url.get_path().length()); m_file_addr.sun_path[sizeof(m_file_addr.sun_path) - 1]= '\0'; m_sa = (sockaddr*)&m_file_addr; m_sa_len = sizeof(struct sockaddr_un); diff --git a/userspace/libsinsp/threadinfo.cpp b/userspace/libsinsp/threadinfo.cpp index 3c30d606f..5709e666e 100644 --- a/userspace/libsinsp/threadinfo.cpp +++ b/userspace/libsinsp/threadinfo.cpp @@ -1180,11 +1180,11 @@ void sinsp_threadinfo::fd_to_scap(scap_fdinfo *dst, sinsp_fdinfo_t* src) case SCAP_FD_UNIX_SOCK: dst->info.unix_socket_info.source = src->m_sockinfo.m_unixinfo.m_fields.m_source; dst->info.unix_socket_info.destination = src->m_sockinfo.m_unixinfo.m_fields.m_dest; - strncpy(dst->info.unix_socket_info.fname, src->m_name.c_str(), SCAP_MAX_PATH_SIZE); + strlcpy(dst->info.unix_socket_info.fname, src->m_name.c_str(), SCAP_MAX_PATH_SIZE); break; case SCAP_FD_FILE_V2: dst->info.regularinfo.open_flags = src->m_openflags; - strncpy(dst->info.regularinfo.fname, src->m_name.c_str(), SCAP_MAX_PATH_SIZE); + strlcpy(dst->info.regularinfo.fname, src->m_name.c_str(), SCAP_MAX_PATH_SIZE); dst->info.regularinfo.dev = src->m_dev; dst->info.regularinfo.mount_id = src->m_mount_id; break; @@ -1198,7 +1198,7 @@ void sinsp_threadinfo::fd_to_scap(scap_fdinfo *dst, sinsp_fdinfo_t* src) case SCAP_FD_INOTIFY: case SCAP_FD_TIMERFD: case SCAP_FD_NETLINK: - strncpy(dst->info.fname, src->m_name.c_str(), SCAP_MAX_PATH_SIZE); + strlcpy(dst->info.fname, src->m_name.c_str(), SCAP_MAX_PATH_SIZE); break; default: ASSERT(false); diff --git a/userspace/libsinsp/utils.h b/userspace/libsinsp/utils.h index a615bdcc9..28a62c20a 100644 --- a/userspace/libsinsp/utils.h +++ b/userspace/libsinsp/utils.h @@ -28,6 +28,7 @@ limitations under the License. #include #include #include "json/json.h" +#include "../common/strlcpy.h" class sinsp_evttables; typedef union _sinsp_sockinfo sinsp_sockinfo; From ed3b0e259679cfd83be0bc41e91f2c5f24d36e26 Mon Sep 17 00:00:00 2001 From: Luca Guerra Date: Fri, 22 Oct 2021 12:12:14 +0000 Subject: [PATCH 045/148] libsinsp, libscap: update strlcpy() use, remove redundant checks Signed-off-by: Luca Guerra --- userspace/libscap/scap.c | 3 +-- userspace/libscap/scap_fds.c | 10 +++++----- userspace/libscap/scap_iflist.c | 4 ++-- userspace/libsinsp/filter.cpp | 8 +++----- userspace/libsinsp/logger.cpp | 1 - userspace/libsinsp/sinsp.cpp | 2 +- userspace/libsinsp/sinsp_curl.cpp | 3 +-- userspace/libsinsp/socket_handler.h | 5 ++--- userspace/libsinsp/threadinfo.cpp | 6 +++--- 9 files changed, 18 insertions(+), 24 deletions(-) diff --git a/userspace/libscap/scap.c b/userspace/libscap/scap.c index c3cb7536a..a06a43211 100644 --- a/userspace/libscap/scap.c +++ b/userspace/libscap/scap.c @@ -2554,8 +2554,7 @@ const char* scap_get_host_root() static char env_str[SCAP_MAX_PATH_SIZE + 1]; static bool inited = false; if (! inited) { - strlcpy(env_str, p ? p : "", SCAP_MAX_PATH_SIZE); - env_str[SCAP_MAX_PATH_SIZE] = '\0'; + strlcpy(env_str, p ? p : "", sizeof(env_str)); inited = true; } diff --git a/userspace/libscap/scap_fds.c b/userspace/libscap/scap_fds.c index 2358e8855..628577305 100644 --- a/userspace/libscap/scap_fds.c +++ b/userspace/libscap/scap_fds.c @@ -751,7 +751,7 @@ int32_t scap_fd_handle_pipe(scap_t *handle, char *fname, scap_threadinfo *tinfo, } ino = sb.st_ino; } - strlcpy(fdi->info.fname, link_name, SCAP_MAX_PATH_SIZE); + strlcpy(fdi->info.fname, link_name, sizeof(fdi->info.fname)); fdi->ino = ino; return scap_add_fd_to_proc_table(handle, tinfo, fdi, error); @@ -977,11 +977,11 @@ int32_t scap_fd_handle_regular_file(scap_t *handle, char *fname, scap_threadinfo else if(fdi->type == SCAP_FD_FILE_V2) { scap_fd_flags_file(handle, fdi, procdir); - strlcpy(fdi->info.regularinfo.fname, link_name, SCAP_MAX_PATH_SIZE); + strlcpy(fdi->info.regularinfo.fname, link_name, sizeof(fdi->info.regularinfo.fname)); } else { - strlcpy(fdi->info.fname, link_name, SCAP_MAX_PATH_SIZE); + strlcpy(fdi->info.fname, link_name, sizeof(fdi->info.fname)); } return scap_add_fd_to_proc_table(handle, tinfo, fdi, error); @@ -1035,7 +1035,7 @@ int32_t scap_fd_handle_socket(scap_t *handle, char *fname, scap_threadinfo *tinf link_name[r] = '\0'; - strlcpy(fdi->info.fname, link_name, SCAP_MAX_PATH_SIZE); + strlcpy(fdi->info.fname, link_name, sizeof(fdi->info.fname)); // link name for sockets should be of the format socket:[ino] if(1 != sscanf(link_name, "socket:[%"PRIi64"]", &ino)) @@ -1169,7 +1169,7 @@ int32_t scap_fd_read_unix_sockets_from_proc_fs(scap_t *handle, const char* filen token = strtok_r(NULL, delimiters, &scratch); if(NULL != token) { - strlcpy(fdinfo->info.unix_socket_info.fname, token, SCAP_MAX_PATH_SIZE); + strlcpy(fdinfo->info.unix_socket_info.fname, token, sizeof(fdinfo->info.unix_socket_info.fname)); } else { diff --git a/userspace/libscap/scap_iflist.c b/userspace/libscap/scap_iflist.c index 014c65481..a2823597c 100644 --- a/userspace/libscap/scap_iflist.c +++ b/userspace/libscap/scap_iflist.c @@ -169,7 +169,7 @@ int32_t scap_create_iflist(scap_t* handle) #else handle->m_addrlist->v4list[ifcnt4].bcast = 0; #endif - strlcpy(handle->m_addrlist->v4list[ifcnt4].ifname, tempIfAddr->ifa_name, SCAP_MAX_PATH_SIZE); + strlcpy(handle->m_addrlist->v4list[ifcnt4].ifname, tempIfAddr->ifa_name, sizeof(handle->m_addrlist->v4list[ifcnt4].ifname)); handle->m_addrlist->v4list[ifcnt4].ifnamelen = strlen(tempIfAddr->ifa_name); handle->m_addrlist->v4list[ifcnt4].linkspeed = 0; @@ -211,7 +211,7 @@ int32_t scap_create_iflist(scap_t* handle) handle->m_addrlist->v4list[ifcnt4].bcast = 0; #endif - strlcpy(handle->m_addrlist->v6list[ifcnt6].ifname, tempIfAddr->ifa_name, SCAP_MAX_PATH_SIZE); + strlcpy(handle->m_addrlist->v6list[ifcnt6].ifname, tempIfAddr->ifa_name, sizeof(handle->m_addrlist->v6list[ifcnt6].ifname)); handle->m_addrlist->v6list[ifcnt6].ifnamelen = strlen(tempIfAddr->ifa_name); handle->m_addrlist->v6list[ifcnt6].linkspeed = 0; diff --git a/userspace/libsinsp/filter.cpp b/userspace/libsinsp/filter.cpp index e3ed1dfaa..2a01a5171 100644 --- a/userspace/libsinsp/filter.cpp +++ b/userspace/libsinsp/filter.cpp @@ -1008,16 +1008,14 @@ char* sinsp_filter_check::rawval_to_string(uint8_t* rawval, return m_getpropertystr_storage; case PT_IPV6ADDR: { - char address[100]; + char address[INET6_ADDRSTRLEN]; - if(NULL == inet_ntop(AF_INET6, rawval, address, 100)) + if(NULL == inet_ntop(AF_INET6, rawval, address, INET6_ADDRSTRLEN)) { strcpy(address, ""); } - strlcpy(m_getpropertystr_storage, - address, - 100); + strlcpy(m_getpropertystr_storage, address, sizeof(m_getpropertystr_storage)); return m_getpropertystr_storage; } diff --git a/userspace/libsinsp/logger.cpp b/userspace/libsinsp/logger.cpp index 80240d344..e8e7b9315 100644 --- a/userspace/libsinsp/logger.cpp +++ b/userspace/libsinsp/logger.cpp @@ -181,7 +181,6 @@ void sinsp_logger::log(std::string msg, const severity sev) { char sev_buf[ENCODE_LEN + 1]; strlcpy(sev_buf, encode_severity(sev), sizeof(sev_buf)); - sev_buf[sizeof(sev_buf) - 1] = 0; msg.insert(0, sev_buf); } diff --git a/userspace/libsinsp/sinsp.cpp b/userspace/libsinsp/sinsp.cpp index 82347fc6a..58d460f53 100644 --- a/userspace/libsinsp/sinsp.cpp +++ b/userspace/libsinsp/sinsp.cpp @@ -589,7 +589,7 @@ int64_t sinsp::get_file_size(const std::string& fname, char *error) } #endif if(errdesc.empty()) errdesc = get_error_desc(err_str); - strlcpy(error, errdesc.c_str(), errdesc.size() > SCAP_LASTERR_SIZE ? SCAP_LASTERR_SIZE : errdesc.size()); + strlcpy(error, errdesc.c_str(), SCAP_LASTERR_SIZE); return -1; } diff --git a/userspace/libsinsp/sinsp_curl.cpp b/userspace/libsinsp/sinsp_curl.cpp index 7c41a7598..90d858938 100644 --- a/userspace/libsinsp/sinsp_curl.cpp +++ b/userspace/libsinsp/sinsp_curl.cpp @@ -239,8 +239,7 @@ size_t sinsp_curl::header_callback(char *buffer, size_t size, size_t nitems, voi if(sz < CURL_MAX_HTTP_HEADER) { g_logger.log("HTTP redirect Location: (" + buf + ')', sinsp_logger::SEV_TRACE); - strlcpy((char*) userdata, buf.data(), sz); - ((char*) userdata)[sz] = 0; + strlcpy((char*) userdata, buf.c_str(), CURL_MAX_HTTP_HEADER); } } return nitems * size; diff --git a/userspace/libsinsp/socket_handler.h b/userspace/libsinsp/socket_handler.h index 441649907..3b7d85cff 100644 --- a/userspace/libsinsp/socket_handler.h +++ b/userspace/libsinsp/socket_handler.h @@ -923,7 +923,7 @@ class socket_data_handler std::memset(buf, 0, size); int pass_len = static_cast(strlen((char*)pass)); if(size < (pass_len) + 1) { return 0; } - strlcpy(buf, (const char*)pass, pass_len); + strlcpy(buf, (const char*)pass, size); return pass_len; } return 0; @@ -1382,8 +1382,7 @@ class socket_data_handler throw sinsp_exception("Invalid address (too long): [" + m_url.get_path() + ']'); } m_file_addr.sun_family = AF_UNIX; - strlcpy(m_file_addr.sun_path, m_url.get_path().c_str(), m_url.get_path().length()); - m_file_addr.sun_path[sizeof(m_file_addr.sun_path) - 1]= '\0'; + strlcpy(m_file_addr.sun_path, m_url.get_path().c_str(), sizeof(m_file_addr.sun_path)); m_sa = (sockaddr*)&m_file_addr; m_sa_len = sizeof(struct sockaddr_un); } diff --git a/userspace/libsinsp/threadinfo.cpp b/userspace/libsinsp/threadinfo.cpp index 5709e666e..1bb3498a1 100644 --- a/userspace/libsinsp/threadinfo.cpp +++ b/userspace/libsinsp/threadinfo.cpp @@ -1180,11 +1180,11 @@ void sinsp_threadinfo::fd_to_scap(scap_fdinfo *dst, sinsp_fdinfo_t* src) case SCAP_FD_UNIX_SOCK: dst->info.unix_socket_info.source = src->m_sockinfo.m_unixinfo.m_fields.m_source; dst->info.unix_socket_info.destination = src->m_sockinfo.m_unixinfo.m_fields.m_dest; - strlcpy(dst->info.unix_socket_info.fname, src->m_name.c_str(), SCAP_MAX_PATH_SIZE); + strlcpy(dst->info.unix_socket_info.fname, src->m_name.c_str(), sizeof(dst->info.unix_socket_info.fname)); break; case SCAP_FD_FILE_V2: dst->info.regularinfo.open_flags = src->m_openflags; - strlcpy(dst->info.regularinfo.fname, src->m_name.c_str(), SCAP_MAX_PATH_SIZE); + strlcpy(dst->info.regularinfo.fname, src->m_name.c_str(), sizeof(dst->info.regularinfo.fname)); dst->info.regularinfo.dev = src->m_dev; dst->info.regularinfo.mount_id = src->m_mount_id; break; @@ -1198,7 +1198,7 @@ void sinsp_threadinfo::fd_to_scap(scap_fdinfo *dst, sinsp_fdinfo_t* src) case SCAP_FD_INOTIFY: case SCAP_FD_TIMERFD: case SCAP_FD_NETLINK: - strlcpy(dst->info.fname, src->m_name.c_str(), SCAP_MAX_PATH_SIZE); + strlcpy(dst->info.fname, src->m_name.c_str(), sizeof(dst->info.fname)); break; default: ASSERT(false); From b62ce64fecbc82528fb00e1879eb3faef23accc9 Mon Sep 17 00:00:00 2001 From: Jason Dellaluce Date: Tue, 26 Oct 2021 14:57:45 +0000 Subject: [PATCH 046/148] update: adding field_id to ss_plugin_extract_field This enables plugin developers to implement more efficient fields lookups during extraction. Co-authored-by: Leonardo Grasso Signed-off-by: Jason Dellaluce --- userspace/libscap/plugin_info.h | 1 + userspace/libsinsp/plugin.cpp | 2 ++ userspace/libsinsp/plugin.h | 1 + 3 files changed, 4 insertions(+) diff --git a/userspace/libscap/plugin_info.h b/userspace/libscap/plugin_info.h index f991abf4e..f64d463c3 100644 --- a/userspace/libscap/plugin_info.h +++ b/userspace/libscap/plugin_info.h @@ -117,6 +117,7 @@ typedef struct ss_plugin_event typedef struct ss_plugin_extract_field { + uint32_t field_id; const char *field; const char *arg; uint32_t ftype; diff --git a/userspace/libsinsp/plugin.cpp b/userspace/libsinsp/plugin.cpp index 2fb2bd3cb..ba964415a 100755 --- a/userspace/libsinsp/plugin.cpp +++ b/userspace/libsinsp/plugin.cpp @@ -277,6 +277,7 @@ class sinsp_filter_check_plugin : public sinsp_filter_check pevt.ts = evt->get_ts(); sinsp_plugin::ext_field field; + field.field_id = m_field_id; field.field = m_info.m_fields[m_field_id].m_name; if(m_arg != NULL) { @@ -611,6 +612,7 @@ bool sinsp_plugin::extract_field(ss_plugin_event &evt, sinsp_plugin::ext_field & uint32_t num_fields = 1; ss_plugin_extract_field efield; + efield.field_id = field.field_id; efield.field = field.field.c_str(); efield.arg = field.arg.c_str(); efield.ftype = field.ftype; diff --git a/userspace/libsinsp/plugin.h b/userspace/libsinsp/plugin.h index 626952a22..4897a00d9 100755 --- a/userspace/libsinsp/plugin.h +++ b/userspace/libsinsp/plugin.h @@ -65,6 +65,7 @@ class sinsp_plugin // Similar to struct ss_plugin_extract_field, but with c++ // types to avoid having to track memory allocations. struct ext_field { + uint32_t field_id; std::string field; std::string arg; uint32_t ftype; From e326ed9af9374a7f7294f600b36474edd3006b79 Mon Sep 17 00:00:00 2001 From: Jason Dellaluce Date: Tue, 26 Oct 2021 15:00:24 +0000 Subject: [PATCH 047/148] refactor: remove free_mem and avoid freeing plugin-allocated memory This implements the newly proposed memory ownership model for which the plugin is totally responsible of both allocating and deallocating its own memory. Co-authored-by: Leonardo Grasso Signed-off-by: Jason Dellaluce --- userspace/libscap/scap.c | 30 ------------------------------ userspace/libsinsp/plugin.cpp | 15 --------------- userspace/libsinsp/plugin.h | 1 - 3 files changed, 46 deletions(-) diff --git a/userspace/libscap/scap.c b/userspace/libscap/scap.c index a06a43211..e157aa005 100644 --- a/userspace/libscap/scap.c +++ b/userspace/libscap/scap.c @@ -115,20 +115,6 @@ static int32_t copy_comms(scap_t *handle, const char **suppressed_comms) return SCAP_SUCCESS; } -static void scap_free_plugin_batch_state(scap_t* handle) -{ - for(uint32_t i = 0; i < handle->m_input_plugin_batch_nevts; i++) - { - handle->m_input_plugin->free_mem(handle->m_input_plugin_batch_evts[i].data); - } - - handle->m_input_plugin_batch_nevts = 0; - handle->m_input_plugin_batch_idx = 0; - - handle->m_input_plugin->free_mem(handle->m_input_plugin_batch_evts); - handle->m_input_plugin_batch_evts = NULL; -} - #if !defined(HAS_CAPTURE) || defined(CYGWING_AGENT) || defined(_WIN32) scap_t* scap_open_live_int(char *error, int32_t *rc, proc_entry_callback proc_callback, @@ -1033,7 +1019,6 @@ scap_t* scap_open_plugin_int(char *error, int32_t *rc, source_plugin_info* input { char *errstr = handle->m_input_plugin->get_last_error(handle->m_input_plugin->state); snprintf(error, SCAP_LASTERR_SIZE, "%s", errstr); - handle->m_input_plugin->free_mem(errstr); scap_close(handle); return NULL; } @@ -1207,10 +1192,7 @@ void scap_close(scap_t* handle) else if(handle->m_mode == SCAP_MODE_PLUGIN) { handle->m_input_plugin->close(handle->m_input_plugin->state, handle->m_input_plugin->handle); - scap_free_plugin_batch_state(handle); handle->m_input_plugin->handle = NULL; - // name was allocated - handle->m_input_plugin->free_mem(handle->m_input_plugin->name); } #if CYGWING_AGENT || _WIN32 @@ -1747,15 +1729,12 @@ static int32_t scap_next_plugin(scap_t* handle, OUT scap_evt** pevent, OUT uint1 { if(handle->m_input_plugin_batch_idx >= handle->m_input_plugin_batch_nevts) { - scap_free_plugin_batch_state(handle); - if(handle->m_input_plugin_last_batch_res != SS_PLUGIN_SUCCESS) { if(handle->m_input_plugin_last_batch_res != SCAP_TIMEOUT && handle->m_input_plugin_last_batch_res != SCAP_EOF) { char *errstr = handle->m_input_plugin->get_last_error(handle->m_input_plugin->state); snprintf(handle->m_lasterr, SCAP_LASTERR_SIZE, "%s", errstr); - handle->m_input_plugin->free_mem(errstr); } int32_t tres = handle->m_input_plugin_last_batch_res; handle->m_input_plugin_last_batch_res = SCAP_SUCCESS; @@ -1782,7 +1761,6 @@ static int32_t scap_next_plugin(scap_t* handle, OUT scap_evt** pevent, OUT uint1 { char *errstr = handle->m_input_plugin->get_last_error(handle->m_input_plugin->state); snprintf(handle->m_lasterr, SCAP_LASTERR_SIZE, "%s", errstr); - handle->m_input_plugin->free_mem(errstr); } return handle->m_input_plugin_last_batch_res; } @@ -1811,7 +1789,6 @@ static int32_t scap_next_plugin(scap_t* handle, OUT scap_evt** pevent, OUT uint1 { char *errstr = handle->m_input_plugin->get_last_error(handle->m_input_plugin->state); snprintf(handle->m_lasterr, SCAP_LASTERR_SIZE, "%s", errstr); - handle->m_input_plugin->free_mem(errstr); } return plugin_rc_to_scap_rc(plugin_res); } @@ -1852,13 +1829,6 @@ static int32_t scap_next_plugin(scap_t* handle, OUT scap_evt** pevent, OUT uint1 memcpy(buf, plugin_evt->data, plugin_evt->datalen); - if(should_free_plugin_evt) - { - handle->m_input_plugin->free_mem(plugin_evt->data); - plugin_evt->data = NULL; - handle->m_input_plugin->free_mem(plugin_evt); - } - if(plugin_evt->ts != UINT64_MAX) { evt->ts = plugin_evt->ts; diff --git a/userspace/libsinsp/plugin.cpp b/userspace/libsinsp/plugin.cpp index ba964415a..3826bd173 100755 --- a/userspace/libsinsp/plugin.cpp +++ b/userspace/libsinsp/plugin.cpp @@ -402,16 +402,6 @@ std::shared_ptr sinsp_plugin::create_plugin(string &filepath, char return ret; } - // Get the plugin's free() function, and return an error if it - // doesn't exist. - void (*free_mem)(void *ptr); - *(void **) (&free_mem) = getsym(handle, "plugin_free_mem", errstr); - if(free_mem == NULL) - { - errstr = string("Could not resolve plugin_free_mem function"); - return ret; - } - // Before doing anything else, check the required api // version. If it doesn't match, return an error. @@ -428,7 +418,6 @@ std::shared_ptr sinsp_plugin::create_plugin(string &filepath, char char *version_cstr = get_required_api_version(); std::string version_str = version_cstr; - free_mem(version_cstr); version v(version_str); if(!v.m_valid) { @@ -672,7 +661,6 @@ std::string sinsp_plugin::str_from_alloc_charbuf(char *charbuf) if(charbuf != NULL) { str = charbuf; - m_plugin_info.free_mem(charbuf); } return str; @@ -682,7 +670,6 @@ bool sinsp_plugin::resolve_dylib_symbols(void *handle, std::string &errstr) { // Some functions are required and return false if not found. if((*(void **) (&(m_plugin_info.get_required_api_version)) = getsym(handle, "plugin_get_required_api_version", errstr)) == NULL || - (*(void **) (&(m_plugin_info.free_mem)) = getsym(handle, "plugin_free_mem", errstr)) == NULL || (*(void **) (&(m_plugin_info.get_last_error)) = getsym(handle, "plugin_get_last_error", errstr)) == NULL || (*(void **) (&(m_plugin_info.get_name)) = getsym(handle, "plugin_get_name", errstr)) == NULL || (*(void **) (&(m_plugin_info.get_description)) = getsym(handle, "plugin_get_description", errstr)) == NULL || @@ -920,7 +907,6 @@ bool sinsp_source_plugin::resolve_dylib_symbols(void *handle, std::string &errst // // Some functions are required and return false if not found. if((*(void **) (&(m_source_plugin_info.get_required_api_version)) = getsym(handle, "plugin_get_required_api_version", errstr)) == NULL || - (*(void **) (&(m_source_plugin_info.free_mem)) = getsym(handle, "plugin_free_mem", errstr)) == NULL || (*(void **) (&(m_source_plugin_info.init)) = getsym(handle, "plugin_init", errstr)) == NULL || (*(void **) (&(m_source_plugin_info.destroy)) = getsym(handle, "plugin_destroy", errstr)) == NULL || (*(void **) (&(m_source_plugin_info.get_last_error)) = getsym(handle, "plugin_get_last_error", errstr)) == NULL || @@ -997,7 +983,6 @@ bool sinsp_extractor_plugin::resolve_dylib_symbols(void *handle, std::string &er // // Some functions are required and return false if not found. if((*(void **) (&(m_extractor_plugin_info.get_required_api_version)) = getsym(handle, "plugin_get_required_api_version", errstr)) == NULL || - (*(void **) (&(m_extractor_plugin_info.free_mem)) = getsym(handle, "plugin_free_mem", errstr)) == NULL || (*(void **) (&(m_extractor_plugin_info.init)) = getsym(handle, "plugin_init", errstr)) == NULL || (*(void **) (&(m_extractor_plugin_info.destroy)) = getsym(handle, "plugin_destroy", errstr)) == NULL || (*(void **) (&(m_extractor_plugin_info.get_last_error)) = getsym(handle, "plugin_get_last_error", errstr)) == NULL || diff --git a/userspace/libsinsp/plugin.h b/userspace/libsinsp/plugin.h index 4897a00d9..a5fc60315 100755 --- a/userspace/libsinsp/plugin.h +++ b/userspace/libsinsp/plugin.h @@ -136,7 +136,6 @@ class sinsp_plugin // included here as they are called in create_plugin() typedef struct { char* (*get_required_api_version)(); - void (*free_mem)(void *ptr); ss_plugin_t* (*init)(char* config, ss_plugin_rc* rc); void (*destroy)(ss_plugin_t* s); char* (*get_last_error)(ss_plugin_t* s); From 5ac4b34e5f0dd6361bce781e7b0de39741a0605e Mon Sep 17 00:00:00 2001 From: Jason Dellaluce Date: Tue, 26 Oct 2021 15:01:21 +0000 Subject: [PATCH 048/148] update: sync plugin_info.h with new memory ownership specifics Co-authored-by: Leonardo Grasso Signed-off-by: Jason Dellaluce --- userspace/libscap/plugin_info.h | 93 +++++++++++++++++---------------- 1 file changed, 48 insertions(+), 45 deletions(-) diff --git a/userspace/libscap/plugin_info.h b/userspace/libscap/plugin_info.h index f64d463c3..1e492481e 100644 --- a/userspace/libscap/plugin_info.h +++ b/userspace/libscap/plugin_info.h @@ -74,9 +74,7 @@ typedef enum ss_plugin_rc // below in next()/next_batch(). // - evtnum: incremented for each event returned. Might not be contiguous. // - data: pointer to a memory buffer pointer. The plugin will set it -// to point to the memory containing the next event. Once returned, -// the memory is owned by the plugin framework and will be freed via -// a call to plugin_free_mem(). +// to point to the memory containing the next event. // - datalen: pointer to a 32bit integer. The plugin will set it the size of the // buffer pointed by data. // - ts: the event timestamp, in nanoseconds since the epoch. @@ -96,6 +94,8 @@ typedef struct ss_plugin_event // Used in extract_fields functions below to receive a field/arg // pair and return an extracted value. +// field_id: id of the field, as of its index in the list of +// fields specified by the plugin. // field: the field name. // arg: the field argument, if an argument has been specified // for the field, otherwise it's NULL. @@ -109,9 +109,8 @@ typedef struct ss_plugin_event // - field_present: set to true if the event has a meaningful // extracted value for the provided field, false otherwise // - res_str: if the corresponding field was type==string, this should be -// filled in with the string value. The string should be allocated by -// the plugin using malloc()/similar and will be free()d by the plugin -// framework by calling plugin_free_mem(). +// filled in with the string value. The string must be allocated and set +// by the plugin. // - res_u64: if the corresponding field was type==uint64, this should be // filled in with the uint64 value. @@ -148,7 +147,7 @@ typedef void ss_instance_t; // // The structs below define the functions and arguments for source and -// exctractor plugins. The structs are used by the plugin framework to +// extractor plugins. The structs are used by the plugin framework to // load and interface with plugins. // // From the perspective of the plugin, each function below should be @@ -156,10 +155,20 @@ typedef void ss_instance_t; // function, adding a prefix "plugin_" to the function name // (e.g. plugin_get_required_api_version, plugin_init, etc.) // -// NOTE: For all functions below that return a char */struct *, the memory -// pointed to by the char */struct * must be allocated by the plugin using -// malloc()/similar and should be freed by the caller using plugin_free_mem(). +// Plugins are totally responsible of both allocating and deallocating memory. +// Plugins have the guarantee that they can safely deallocate memory in +// these cases: +// - During close(), for all the memory allocated in the context of a plugin +// instance after open(). +// - During destroy(), for all the memory allocated by the plugin, as it stops +// being executed. +// - During subsequent calls to the same function, for all the exported +// functions returning memory pointers. // +// Plugins must not free memory passed in by the framework (i.e. function input +// parameters) if not corresponding to plugin-allocated memory in the +// cases above. Plugins can safely use the passed memory during the execution +// of the exported functions. // // Interface for a sinsp/scap source plugin. @@ -178,12 +187,6 @@ typedef struct // char* (*get_required_api_version)(); // - // The plugin framework will call this function to free any - // memory allocated by the plugin and returned to the - // framework. This includes return values from get_type()/get_name()/..., - // get_last_error(), event structs returned in next_batch(), etc. - void (*free_mem)(void *ptr); - // // Return the plugin type. // Required: yes // Should return TYPE_SOURCE_PLUGIN. It still makes sense to @@ -262,7 +265,7 @@ typedef struct // Return a string describing the events generated by this source plugin. // Required: yes // Example event sources would be strings like "syscall", - // "k8s_audit", etc. The source can be used by extractor + // "k8s_audit", etc. The source can be used by extractor // plugins to filter the events they receive. // char* (*get_event_source)(); @@ -314,12 +317,12 @@ typedef struct // Arguments: // - s: the plugin context, returned by init(). Can be NULL. // - h: the capture context, returned by open(). Can be NULL. - // - // - evt: pointer to a ss_plugin_event pointer. The plugin should - // allocate a ss_plugin_event struct using malloc(), as well as - // allocate the data buffer within the ss_plugin_event struct. - // Both the struct and data buffer are owned by the plugin framework - // and will free them using plugin_free_mem(). + // - evt: pointer to a ss_plugin_event pointer. The plugin must + // allocate a ss_plugin_event struct and the data buffer within + // the ss_plugin_event struct. + // Once returned, both the struct and data buffer are owned by + // the framework until the next call to next(), next_batch(), + // or close(). // // Return value: the status of the operation (e.g. SS_PLUGIN_SUCCESS=0, SS_PLUGIN_FAILURE=1, // SS_PLUGIN_TIMEOUT=-1) @@ -356,11 +359,16 @@ typedef struct // Extract one or more a filter field values from an event. // Required: no // Arguments: - // - evt: an event struct returned by a call to next()/batch_next(). + // - evt: an event struct produced by a call to next()/batch_next(). + // This is allocated by the framework, and it is not guaranteed + // that the event struct pointer is the same returned by the last + // next()/batch_next() call. // - num_fields: the length of the fields array. - // - fields: an array of ss_plugin_extract_field structs. Each element contains - // a single field + optional arg, and the corresponding extracted value should - // be in the same struct. + // - fields: an array of ss_plugin_extract_field structs. Each entry + // contains a single field + optional argument as input, and the corresponding + // extracted value as output. Memory pointers set as output must be allocated + // by the plugin. + // // Return value: An ss_plugin_rc with values SS_PLUGIN_SUCCESS or SS_PLUGIN_FAILURE. // ss_plugin_rc (*extract_fields)(ss_plugin_t *s, const ss_plugin_event *evt, uint32_t num_fields, ss_plugin_extract_field *fields); @@ -369,15 +377,15 @@ typedef struct // batching the calls to next(). // On success: // - nevts will be filled in with the number of events. - // - evts: pointer to an ss_plugin_event pointer. The plugin should - // allocate an array of contiguous ss_plugin_event structs using malloc(), - // as well as allocate each data buffer within each ss_plugin_event - // struct using malloc(). Both the array of structs and each data buffer are - // owned by the plugin framework and will free them using plugin_free_mem(). + // - evts: pointer to an ss_plugin_event pointer. The plugin must + // allocate an array of contiguous ss_plugin_event structs + // and each data buffer within each ss_plugin_event struct. + // Both the array of structs and each data buffer are owned by + // the framework until the next call to next(), next_batch(), + // or close(). // Required: no // ss_plugin_rc (*next_batch)(ss_plugin_t* s, ss_instance_t* h, uint32_t *nevts, ss_plugin_event **evts); - // // The following members are PRIVATE for the engine and should not be touched. // @@ -404,12 +412,6 @@ typedef struct // char* (*get_required_api_version)(); // - // The plugin framework will call this function to free any - // memory allocated by the plugin and returned to the - // framework. This includes return values from get_type()/get_name()/..., - // get_last_error(), strings in extract_fields(), etc. - void (*free_mem)(void *ptr); - // // Return the plugin type. // Required: yes // Should return TYPE_EXTRACTOR_PLUGIN. It still makes sense to @@ -497,17 +499,18 @@ typedef struct char* (*get_fields)(); // // Extract one or more a filter field values from an event. - // Required: no + // Required: yes // Arguments: - // - evt: an event struct returned by a call to next()/batch_next(). + // - evt: an event struct provided by the framework. // - num_fields: the length of the fields array. - // - fields: an array of ss_plugin_extract_field structs. Each element contains - // a single field + optional arg, and the corresponding extracted value should - // be in the same struct. + // - fields: an array of ss_plugin_extract_field structs. Each entry + // contains a single field + optional argument as input, and the corresponding + // extracted value as output. Memory pointers set as output must be allocated + // by the plugin. + // // Return value: An integer with values SS_PLUGIN_SUCCESS or SS_PLUGIN_FAILURE. // ss_plugin_rc (*extract_fields)(ss_plugin_t *s, const ss_plugin_event *evt, uint32_t num_fields, ss_plugin_extract_field *fields); - // // The following members are PRIVATE for the engine and should not be touched. // From 48e61a59d4c1890fe12948cd9ff7767ef1aec912 Mon Sep 17 00:00:00 2001 From: Jason Dellaluce Date: Tue, 26 Oct 2021 15:01:51 +0000 Subject: [PATCH 049/148] update: bump plugin api version (to 0.2.0) Co-authored-by: Leonardo Grasso Signed-off-by: Jason Dellaluce --- userspace/libscap/plugin_info.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/userspace/libscap/plugin_info.h b/userspace/libscap/plugin_info.h index 1e492481e..470bbc212 100644 --- a/userspace/libscap/plugin_info.h +++ b/userspace/libscap/plugin_info.h @@ -28,7 +28,7 @@ limitations under the License. // API versions of this plugin engine // #define PLUGIN_API_VERSION_MAJOR 0 -#define PLUGIN_API_VERSION_MINOR 1 +#define PLUGIN_API_VERSION_MINOR 2 #define PLUGIN_API_VERSION_PATCH 0 // From 804d3d1c4662975f77adc2250b2cef86adb51236 Mon Sep 17 00:00:00 2001 From: Jason Dellaluce Date: Tue, 26 Oct 2021 16:28:10 +0000 Subject: [PATCH 050/148] update: remove next symbol As discussed with other contributors and proposed by @mstemm, we remove the next exported symbol to present an easier interface to plugin authors. Co-authored-by: Mark Stemm Co-authored-by: Leonardo Grasso Signed-off-by: Jason Dellaluce --- userspace/libscap/plugin_info.h | 41 ++++----------- userspace/libscap/scap.c | 90 +++++++++++++-------------------- userspace/libsinsp/plugin.cpp | 3 +- userspace/libsinsp/plugin.h | 2 +- 4 files changed, 47 insertions(+), 89 deletions(-) diff --git a/userspace/libscap/plugin_info.h b/userspace/libscap/plugin_info.h index 470bbc212..c2e0043cf 100644 --- a/userspace/libscap/plugin_info.h +++ b/userspace/libscap/plugin_info.h @@ -59,7 +59,7 @@ typedef enum ss_plugin_field_type FTYPE_STRING = 9 }ss_plugin_field_type; -// Values to return from init() / open() / next() / next_batch() / +// Values to return from init() / open() / next_batch() / // extract_fields(). typedef enum ss_plugin_rc { @@ -71,7 +71,7 @@ typedef enum ss_plugin_rc } ss_plugin_rc; // This struct represents an event returned by the plugin, and is used -// below in next()/next_batch(). +// below in next_batch(). // - evtnum: incremented for each event returned. Might not be contiguous. // - data: pointer to a memory buffer pointer. The plugin will set it // to point to the memory containing the next event. @@ -83,7 +83,7 @@ typedef enum ss_plugin_rc // // Note: event numbers are assigned by the plugin // framework. Therefore, there isn't any need to fill in evtnum when -// returning an event via plugin_next/plugin_next_batch. It will be ignored. +// returning an event via plugin_next_batch. It will be ignored. typedef struct ss_plugin_event { uint64_t evtnum; @@ -299,7 +299,7 @@ typedef struct // itsef // - rc: pointer to a ss_plugin_rc that will contain the open result, // as a SS_PLUGIN_* value (e.g. SS_PLUGIN_SUCCESS=0, SS_PLUGIN_FAILURE=1) - // Return value: a pointer to the open context that will be passed to next(), + // Return value: a pointer to the open context that will be passed to next_batch(), // close(), event_to_string() and extract_fields. // ss_instance_t* (*open)(ss_plugin_t* s, char* params, ss_plugin_rc* rc); @@ -312,23 +312,6 @@ typedef struct // void (*close)(ss_plugin_t* s, ss_instance_t* h); // - // Return the next event. - // Required: yes - // Arguments: - // - s: the plugin context, returned by init(). Can be NULL. - // - h: the capture context, returned by open(). Can be NULL. - // - evt: pointer to a ss_plugin_event pointer. The plugin must - // allocate a ss_plugin_event struct and the data buffer within - // the ss_plugin_event struct. - // Once returned, both the struct and data buffer are owned by - // the framework until the next call to next(), next_batch(), - // or close(). - // - // Return value: the status of the operation (e.g. SS_PLUGIN_SUCCESS=0, SS_PLUGIN_FAILURE=1, - // SS_PLUGIN_TIMEOUT=-1) - // - ss_plugin_rc (*next)(ss_plugin_t* s, ss_instance_t* h, ss_plugin_event **evt); - // // Return the read progress. // Required: no // Arguments: @@ -349,8 +332,8 @@ typedef struct // Return a text representation of an event generated by this source plugin. // Required: yes // Arguments: - // - data: the buffer from an event produced by next(). - // - datalen: the length of the buffer from an event produced by next(). + // - data: the buffer from an event produced by next_batch(). + // - datalen: the length of the buffer from an event produced by next_batch(). // Return value: the text representation of the event. This is used, for example, // by sysdig to print a line for the given event. // @@ -359,10 +342,10 @@ typedef struct // Extract one or more a filter field values from an event. // Required: no // Arguments: - // - evt: an event struct produced by a call to next()/batch_next(). + // - evt: an event struct produced by a call to batch_next(). // This is allocated by the framework, and it is not guaranteed // that the event struct pointer is the same returned by the last - // next()/batch_next() call. + // batch_next() call. // - num_fields: the length of the fields array. // - fields: an array of ss_plugin_extract_field structs. Each entry // contains a single field + optional argument as input, and the corresponding @@ -373,17 +356,15 @@ typedef struct // ss_plugin_rc (*extract_fields)(ss_plugin_t *s, const ss_plugin_event *evt, uint32_t num_fields, ss_plugin_extract_field *fields); // - // This is an optional, internal, function used to speed up event capture by - // batching the calls to next(). + // Return the next batch of events. // On success: // - nevts will be filled in with the number of events. // - evts: pointer to an ss_plugin_event pointer. The plugin must // allocate an array of contiguous ss_plugin_event structs // and each data buffer within each ss_plugin_event struct. // Both the array of structs and each data buffer are owned by - // the framework until the next call to next(), next_batch(), - // or close(). - // Required: no + // the framework until the next call to next_batch() or close(). + // Required: yes // ss_plugin_rc (*next_batch)(ss_plugin_t* s, ss_instance_t* h, uint32_t *nevts, ss_plugin_event **evts); // diff --git a/userspace/libscap/scap.c b/userspace/libscap/scap.c index e157aa005..80c76533f 100644 --- a/userspace/libscap/scap.c +++ b/userspace/libscap/scap.c @@ -1723,78 +1723,56 @@ static int32_t scap_next_plugin(scap_t* handle, OUT scap_evt** pevent, OUT uint1 { ss_plugin_event *plugin_evt; int32_t res = SCAP_FAILURE; - bool should_free_plugin_evt = false; - if(handle->m_input_plugin->next_batch != NULL) + if(handle->m_input_plugin_batch_idx >= handle->m_input_plugin_batch_nevts) { - if(handle->m_input_plugin_batch_idx >= handle->m_input_plugin_batch_nevts) + if(handle->m_input_plugin_last_batch_res != SS_PLUGIN_SUCCESS) { - if(handle->m_input_plugin_last_batch_res != SS_PLUGIN_SUCCESS) + if(handle->m_input_plugin_last_batch_res != SCAP_TIMEOUT && handle->m_input_plugin_last_batch_res != SCAP_EOF) { - if(handle->m_input_plugin_last_batch_res != SCAP_TIMEOUT && handle->m_input_plugin_last_batch_res != SCAP_EOF) - { - char *errstr = handle->m_input_plugin->get_last_error(handle->m_input_plugin->state); - snprintf(handle->m_lasterr, SCAP_LASTERR_SIZE, "%s", errstr); - } - int32_t tres = handle->m_input_plugin_last_batch_res; - handle->m_input_plugin_last_batch_res = SCAP_SUCCESS; - return tres; + char *errstr = handle->m_input_plugin->get_last_error(handle->m_input_plugin->state); + snprintf(handle->m_lasterr, SCAP_LASTERR_SIZE, "%s", errstr); } + int32_t tres = handle->m_input_plugin_last_batch_res; + handle->m_input_plugin_last_batch_res = SCAP_SUCCESS; + return tres; + } - int32_t plugin_res = handle->m_input_plugin->next_batch(handle->m_input_plugin->state, - handle->m_input_plugin->handle, - &(handle->m_input_plugin_batch_nevts), - &(handle->m_input_plugin_batch_evts)); - handle->m_input_plugin_last_batch_res = plugin_rc_to_scap_rc(plugin_res); - - if(handle->m_input_plugin_batch_nevts == 0) + int32_t plugin_res = handle->m_input_plugin->next_batch(handle->m_input_plugin->state, + handle->m_input_plugin->handle, + &(handle->m_input_plugin_batch_nevts), + &(handle->m_input_plugin_batch_evts)); + handle->m_input_plugin_last_batch_res = plugin_rc_to_scap_rc(plugin_res); + + if(handle->m_input_plugin_batch_nevts == 0) + { + if(handle->m_input_plugin_last_batch_res == SCAP_SUCCESS) { - if(handle->m_input_plugin_last_batch_res == SCAP_SUCCESS) - { - snprintf(handle->m_lasterr, SCAP_LASTERR_SIZE, "unexpected 0 size event returned by plugin %s", handle->m_input_plugin->name); - ASSERT(false); - return SCAP_FAILURE; - } - else + snprintf(handle->m_lasterr, SCAP_LASTERR_SIZE, "unexpected 0 size event returned by plugin %s", handle->m_input_plugin->name); + ASSERT(false); + return SCAP_FAILURE; + } + else + { + if(handle->m_input_plugin_last_batch_res != SCAP_TIMEOUT && handle->m_input_plugin_last_batch_res != SCAP_EOF) { - if(handle->m_input_plugin_last_batch_res != SCAP_TIMEOUT && handle->m_input_plugin_last_batch_res != SCAP_EOF) - { - char *errstr = handle->m_input_plugin->get_last_error(handle->m_input_plugin->state); - snprintf(handle->m_lasterr, SCAP_LASTERR_SIZE, "%s", errstr); - } - return handle->m_input_plugin_last_batch_res; + char *errstr = handle->m_input_plugin->get_last_error(handle->m_input_plugin->state); + snprintf(handle->m_lasterr, SCAP_LASTERR_SIZE, "%s", errstr); } + return handle->m_input_plugin_last_batch_res; } - - handle->m_input_plugin_batch_idx = 0; } - uint32_t pos = handle->m_input_plugin_batch_idx; + handle->m_input_plugin_batch_idx = 0; + } - plugin_evt = &(handle->m_input_plugin_batch_evts[pos]); + uint32_t pos = handle->m_input_plugin_batch_idx; - handle->m_input_plugin_batch_idx++; + plugin_evt = &(handle->m_input_plugin_batch_evts[pos]); - res = SCAP_SUCCESS; - } - else - { - should_free_plugin_evt = true; + handle->m_input_plugin_batch_idx++; - ss_plugin_rc plugin_res = handle->m_input_plugin->next(handle->m_input_plugin->state, - handle->m_input_plugin->handle, &plugin_evt); - if(plugin_res != SS_PLUGIN_SUCCESS) - { - if(plugin_res != SS_PLUGIN_TIMEOUT && res != SS_PLUGIN_EOF) - { - char *errstr = handle->m_input_plugin->get_last_error(handle->m_input_plugin->state); - snprintf(handle->m_lasterr, SCAP_LASTERR_SIZE, "%s", errstr); - } - return plugin_rc_to_scap_rc(plugin_res); - } - - res = SCAP_SUCCESS; - } + res = SCAP_SUCCESS; // The numbers are: // - size of plugin id param length (16 bits), holding the value 4 diff --git a/userspace/libsinsp/plugin.cpp b/userspace/libsinsp/plugin.cpp index 3826bd173..411be53e7 100755 --- a/userspace/libsinsp/plugin.cpp +++ b/userspace/libsinsp/plugin.cpp @@ -919,7 +919,7 @@ bool sinsp_source_plugin::resolve_dylib_symbols(void *handle, std::string &errst (*(void **) (&(m_source_plugin_info.get_event_source)) = getsym(handle, "plugin_get_event_source", errstr)) == NULL || (*(void **) (&(m_source_plugin_info.open)) = getsym(handle, "plugin_open", errstr)) == NULL || (*(void **) (&(m_source_plugin_info.close)) = getsym(handle, "plugin_close", errstr)) == NULL || - (*(void **) (&(m_source_plugin_info.next)) = getsym(handle, "plugin_next", errstr)) == NULL || + (*(void **) (&(m_source_plugin_info.next_batch)) = getsym(handle, "plugin_next_batch", errstr)) == NULL || (*(void **) (&(m_source_plugin_info.event_to_string)) = getsym(handle, "plugin_event_to_string", errstr)) == NULL) { return false; @@ -930,7 +930,6 @@ bool sinsp_source_plugin::resolve_dylib_symbols(void *handle, std::string &errst (*(void **) (&m_source_plugin_info.get_progress)) = getsym(handle, "plugin_get_progress", errstr); (*(void **) (&m_source_plugin_info.event_to_string)) = getsym(handle, "plugin_event_to_string", errstr); (*(void **) (&m_source_plugin_info.extract_fields)) = getsym(handle, "plugin_extract_fields", errstr); - (*(void **) (&m_source_plugin_info.next_batch)) = getsym(handle, "plugin_next_batch", errstr); m_id = m_source_plugin_info.get_id(); m_event_source = str_from_alloc_charbuf(m_source_plugin_info.get_event_source()); diff --git a/userspace/libsinsp/plugin.h b/userspace/libsinsp/plugin.h index a5fc60315..2bbdb0f62 100755 --- a/userspace/libsinsp/plugin.h +++ b/userspace/libsinsp/plugin.h @@ -160,7 +160,7 @@ class sinsp_plugin common_plugin_info m_plugin_info; }; -// Note that this doesn't have a next() method, as event generation is +// Note that this doesn't have a next_batch() method, as event generation is // handled at the libscap level. class sinsp_source_plugin : public sinsp_plugin { From 492b66937037ef571679a31e7947085d74c3cb02 Mon Sep 17 00:00:00 2001 From: Jason Dellaluce Date: Wed, 27 Oct 2021 07:46:01 +0000 Subject: [PATCH 051/148] docs: improve plugin exported symbols documentation Co-authored-by: Luca Guerra Co-authored-by: Leonardo Grasso Signed-off-by: Jason Dellaluce --- userspace/libscap/plugin_info.h | 21 +++++++++++++++------ userspace/libscap/scap-int.h | 2 +- 2 files changed, 16 insertions(+), 7 deletions(-) diff --git a/userspace/libscap/plugin_info.h b/userspace/libscap/plugin_info.h index c2e0043cf..7ed929a24 100644 --- a/userspace/libscap/plugin_info.h +++ b/userspace/libscap/plugin_info.h @@ -323,6 +323,9 @@ typedef struct // progress. This might include the progress percentage // combined with additional context added by the plugin. If // NULL, progress_pct should be used. + // The returned memory pointer must be allocated by the plugin + // and must not be deallocated or modified until the next call to + // get_progress(). // NOTE: reporting progress is optional and in some case could be impossible. However, // when possible, it's recommended as it provides valuable information to the // user. @@ -336,21 +339,25 @@ typedef struct // - datalen: the length of the buffer from an event produced by next_batch(). // Return value: the text representation of the event. This is used, for example, // by sysdig to print a line for the given event. + // The returned memory pointer must be allocated by the plugin + // and must not be deallocated or modified until the next call to + // event_to_string(). // char *(*event_to_string)(ss_plugin_t *s, const uint8_t *data, uint32_t datalen); // // Extract one or more a filter field values from an event. // Required: no // Arguments: - // - evt: an event struct produced by a call to batch_next(). + // - evt: an event struct produced by a call to next_batch(). // This is allocated by the framework, and it is not guaranteed // that the event struct pointer is the same returned by the last - // batch_next() call. + // next_batch() call. // - num_fields: the length of the fields array. // - fields: an array of ss_plugin_extract_field structs. Each entry // contains a single field + optional argument as input, and the corresponding // extracted value as output. Memory pointers set as output must be allocated - // by the plugin. + // by the plugin and must not be deallocated or modified until the next + // extract_fields() call. // // Return value: An ss_plugin_rc with values SS_PLUGIN_SUCCESS or SS_PLUGIN_FAILURE. // @@ -362,8 +369,9 @@ typedef struct // - evts: pointer to an ss_plugin_event pointer. The plugin must // allocate an array of contiguous ss_plugin_event structs // and each data buffer within each ss_plugin_event struct. - // Both the array of structs and each data buffer are owned by - // the framework until the next call to next_batch() or close(). + // Memory pointers set as output must be allocated by the plugin + // and must not be deallocated or modified until the next call to + // next_batch() or close(). // Required: yes // ss_plugin_rc (*next_batch)(ss_plugin_t* s, ss_instance_t* h, uint32_t *nevts, ss_plugin_event **evts); @@ -487,7 +495,8 @@ typedef struct // - fields: an array of ss_plugin_extract_field structs. Each entry // contains a single field + optional argument as input, and the corresponding // extracted value as output. Memory pointers set as output must be allocated - // by the plugin. + // by the plugin and must not be deallocated or modified until the next + // extract_fields() call. // // Return value: An integer with values SS_PLUGIN_SUCCESS or SS_PLUGIN_FAILURE. // diff --git a/userspace/libscap/scap-int.h b/userspace/libscap/scap-int.h index a5c30681c..2213a1ae1 100644 --- a/userspace/libscap/scap-int.h +++ b/userspace/libscap/scap-int.h @@ -195,7 +195,7 @@ struct scap // via a call to next(). uint32_t m_input_plugin_batch_idx; - // The return value from the last call to batch_next(). + // The return value from the last call to next_batch(). ss_plugin_rc m_input_plugin_last_batch_res; }; From 1a8bb391e6f526a4837fdb8a0dc718b0cda46416 Mon Sep 17 00:00:00 2001 From: Mark Stemm Date: Tue, 26 Oct 2021 16:49:54 -0700 Subject: [PATCH 052/148] Set event types properly for plugin filterchecks All plugin filterchecks will only work on the plugin event type PPME_PLUGINEVENT_E, so override the default evttypes() method to return that event type only. Signed-off-by: Mark Stemm --- userspace/libsinsp/plugin.cpp | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/userspace/libsinsp/plugin.cpp b/userspace/libsinsp/plugin.cpp index 411be53e7..6e752f65e 100755 --- a/userspace/libsinsp/plugin.cpp +++ b/userspace/libsinsp/plugin.cpp @@ -20,6 +20,7 @@ limitations under the License. #include #include #include +#include #include #endif #include @@ -46,6 +47,8 @@ const filtercheck_field_info sinsp_filter_check_plugininfo_fields[] = {PT_CHARBUF, EPF_NONE, PF_NA, "evt.plugininfo", "if the event comes from a plugin, a summary of the event as formatted by the plugin."}, }; +static std::set s_all_plugin_event_types = {PPME_PLUGINEVENT_E}; + class sinsp_filter_check_plugininfo : public sinsp_filter_check { public: @@ -87,6 +90,11 @@ class sinsp_filter_check_plugininfo : public sinsp_filter_check return new sinsp_filter_check_plugininfo(*this); } + const std::set &evttypes() + { + return s_all_plugin_event_types; + } + uint8_t* extract(sinsp_evt *evt, OUT uint32_t* len, bool sanitize_strings) { // @@ -169,6 +177,11 @@ class sinsp_filter_check_plugin : public sinsp_filter_check { } + const std::set &evttypes() + { + return s_all_plugin_event_types; + } + int32_t parse_field_name(const char* str, bool alloc_state, bool needed_for_filtering) { int32_t res = sinsp_filter_check::parse_field_name(str, alloc_state, needed_for_filtering); From 94914845127895456ff633a080ff7fc1252f4062 Mon Sep 17 00:00:00 2001 From: Mark Stemm Date: Tue, 26 Oct 2021 16:50:40 -0700 Subject: [PATCH 053/148] Fill in some error string when plugin init fails If plugin init fails, return some kind of error string rather than nothing. The init failure is likely before plugin_init completes, so we can't call get_last_error(). Signed-off-by: Mark Stemm --- userspace/libsinsp/plugin.cpp | 1 + 1 file changed, 1 insertion(+) diff --git a/userspace/libsinsp/plugin.cpp b/userspace/libsinsp/plugin.cpp index 6e752f65e..395ccfefa 100755 --- a/userspace/libsinsp/plugin.cpp +++ b/userspace/libsinsp/plugin.cpp @@ -484,6 +484,7 @@ std::shared_ptr sinsp_plugin::create_plugin(string &filepath, char // Initialize the plugin if (!ret->init(config)) { + errstr = string("Could not initialize plugin"); ret = NULL; } From a0a27a9a020f02a1c28ea33196d291d49025171c Mon Sep 17 00:00:00 2001 From: Mark Stemm Date: Tue, 26 Oct 2021 16:51:57 -0700 Subject: [PATCH 054/148] Properly set event type in plugin_infos() Previously the type was checked but not returned in the list of ::info structs. Signed-off-by: Mark Stemm --- userspace/libsinsp/plugin.cpp | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/userspace/libsinsp/plugin.cpp b/userspace/libsinsp/plugin.cpp index 395ccfefa..d228d571a 100755 --- a/userspace/libsinsp/plugin.cpp +++ b/userspace/libsinsp/plugin.cpp @@ -503,8 +503,9 @@ std::list sinsp_plugin::plugin_infos(sinsp* inspector) info.contact = p->contact(); info.plugin_version = p->plugin_version(); info.required_api_version = p->required_api_version(); + info.type = p->type(); - if(p->type() == TYPE_SOURCE_PLUGIN) + if(info.type == TYPE_SOURCE_PLUGIN) { sinsp_source_plugin *sp = static_cast(p.get()); info.id = sp->id(); From ae6df6f9e1410b439eceaa4e807b3e8ac97cba2d Mon Sep 17 00:00:00 2001 From: Federico Di Pierro Date: Fri, 22 Oct 2021 14:31:49 +0200 Subject: [PATCH 055/148] new(userspace/libscap): added new EV(F)_BLOCK_TYPE_V2_LARGE block types, that mark a block type with a 4B argument payload size header. Automatically resize scap handle->m_file_evt_buf to fit largest payload size (starting from 65536, ie: normal syscalls limit). Use 4B size header for plugins payloads. Syscall driven events still use old block type. new(userspace/libsinsp): Use 4B size header for container metadata. Switch various len function-local variables to uint32_t instead of uint16_t to deal with possible new size. Updated event::load_params() to account for new EF_LARGE_PAYLOAD flag. new(driver): added EF_LARGE_PAYLOAD flag. Updated event_table adding EF_LARGE_PAYLOAD flag for PPME_CONTAINER_JSON_E and PPME_PLUGINEVENT_E. Signed-off-by: Federico Di Pierro --- driver/event_table.c | 4 +-- driver/ppm_events_public.h | 3 +- userspace/libscap/scap-int.h | 4 ++- userspace/libscap/scap.c | 31 ++++++++++++------- userspace/libscap/scap.h | 3 +- userspace/libscap/scap_savefile.c | 50 +++++++++++++++++++++++-------- userspace/libscap/scap_savefile.h | 4 +++ userspace/libsinsp/container.cpp | 10 +++---- userspace/libsinsp/event.cpp | 9 ++++-- userspace/libsinsp/event.h | 35 +++++++++++++++++----- 10 files changed, 112 insertions(+), 41 deletions(-) diff --git a/driver/event_table.c b/driver/event_table.c index 2926628fb..d4345ea51 100644 --- a/driver/event_table.c +++ b/driver/event_table.c @@ -284,7 +284,7 @@ const struct ppm_event_info g_event_info[PPM_EVENT_MAX] = { /* PPME_TRACER_X */{ "tracer", EC_OTHER, EF_NONE, 3, { { "id", PT_INT64, PF_DEC }, { "tags", PT_CHARBUFARRAY, PF_NA }, { "args", PT_CHARBUF_PAIR_ARRAY, PF_NA } } }, /* PPME_MESOS_E */{"mesos", EC_INTERNAL, EF_SKIPPARSERESET | EF_MODIFIES_STATE, 1, {{"json", PT_CHARBUF, PF_NA} } }, /* PPME_MESOS_X */{"NA4", EC_SYSTEM, EF_UNUSED, 0}, - /* PPME_CONTAINER_JSON_E */{"container", EC_PROCESS, EF_MODIFIES_STATE, 1, {{"json", PT_CHARBUF, PF_NA} } }, + /* PPME_CONTAINER_JSON_E */{"container", EC_PROCESS, EF_MODIFIES_STATE | EF_LARGE_PAYLOAD, 1, {{"json", PT_CHARBUF, PF_NA} } }, /* PPME_CONTAINER_JSON_X */{"container", EC_PROCESS, EF_UNUSED, 0}, /* PPME_SYSCALL_SETSID_E */{"setsid", EC_PROCESS, EF_MODIFIES_STATE, 0}, /* PPME_SYSCALL_SETSID_X */{"setsid", EC_PROCESS, EF_MODIFIES_STATE, 1, {{"res", PT_PID, PF_DEC} } }, @@ -334,7 +334,7 @@ const struct ppm_event_info g_event_info[PPM_EVENT_MAX] = { /* PPME_SYSCALL_RENAMEAT2_X */{"renameat2", EC_FILE, EF_NONE, 6, {{"res", PT_ERRNO, PF_DEC}, {"olddirfd", PT_FD, PF_DEC}, {"oldpath", PT_FSRELPATH, PF_NA, DIRFD_PARAM(1)}, {"newdirfd", PT_FD, PF_DEC}, {"newpath", PT_FSRELPATH, PF_NA, DIRFD_PARAM(3)}, {"flags", PT_FLAGS32, PF_HEX, renameat2_flags} } }, /* PPME_SYSCALL_USERFAULTFD_E */{"userfaultfd", EC_FILE, EF_CREATES_FD | EF_MODIFIES_STATE, 0}, /* PPME_SYSCALL_USERFAULTFD_X */{"userfaultfd", EC_FILE, EF_CREATES_FD | EF_MODIFIES_STATE, 2, {{"res", PT_ERRNO, PF_DEC}, {"flags", PT_FLAGS32, PF_HEX, file_flags} } }, - /* PPME_PLUGINEVENT_E */{"pluginevent", EC_OTHER, EF_NONE, 2, {{"plugin ID", PT_UINT32, PF_DEC}, {"event_data", PT_BYTEBUF, PF_NA} } }, + /* PPME_PLUGINEVENT_E */{"pluginevent", EC_OTHER, EF_LARGE_PAYLOAD, 2, {{"plugin ID", PT_UINT32, PF_DEC}, {"event_data", PT_BYTEBUF, PF_NA} } }, /* PPME_NA1 */{"pluginevent", EC_OTHER, EF_UNUSED, 0} /* NB: Starting from scap version 1.2, event types will no longer be changed when an event is modified, and the only kind of change permitted for pre-existent events is adding parameters. * New event types are allowed only for new syscalls or new internal events. diff --git a/driver/ppm_events_public.h b/driver/ppm_events_public.h index 2329bcaf9..4924442ee 100644 --- a/driver/ppm_events_public.h +++ b/driver/ppm_events_public.h @@ -1334,7 +1334,8 @@ enum ppm_event_flags { EF_WAITS = (1 << 7), /* This event reads data from an FD. */ EF_SKIPPARSERESET = (1 << 8), /* This event shouldn't pollute the parser lastevent state tracker. */ EF_OLD_VERSION = (1 << 9), /* This event is kept for backward compatibility */ - EF_DROP_SIMPLE_CONS = (1 << 10) /* This event can be skipped by consumers that privilege low overhead to full event capture */ + EF_DROP_SIMPLE_CONS = (1 << 10), /* This event can be skipped by consumers that privilege low overhead to full event capture */ + EF_LARGE_PAYLOAD = (1 << 11), /* This event has a large payload, ie: up to UINT32_MAX bytes. DO NOT USE ON syscalls-driven events!!! */ }; /* diff --git a/userspace/libscap/scap-int.h b/userspace/libscap/scap-int.h index 2213a1ae1..8417d8753 100644 --- a/userspace/libscap/scap-int.h +++ b/userspace/libscap/scap-int.h @@ -124,6 +124,8 @@ struct scap FILE* m_file; #endif char* m_file_evt_buf; + size_t m_file_evt_buf_size; + uint32_t m_last_evt_dump_flags; char m_lasterr[SCAP_LASTERR_SIZE]; @@ -225,7 +227,7 @@ struct scap_ns_socket_list // Misc stuff // #define MEMBER_SIZE(type, member) sizeof(((type *)0)->member) -#define FILE_READ_BUF_SIZE 65536 +#define FILE_READ_BUF_SIZE (1 << 16) // UINT16_MAX + 1, ie: 65536 // // Internal library functions diff --git a/userspace/libscap/scap.c b/userspace/libscap/scap.c index 80c76533f..156f61219 100644 --- a/userspace/libscap/scap.c +++ b/userspace/libscap/scap.c @@ -747,6 +747,7 @@ scap_t* scap_open_offline_int(gzFile gzfile, *rc = SCAP_FAILURE; return NULL; } + handle->m_file_evt_buf_size = FILE_READ_BUF_SIZE; handle->m_file = gzfile; @@ -1774,15 +1775,26 @@ static int32_t scap_next_plugin(scap_t* handle, OUT scap_evt** pevent, OUT uint1 res = SCAP_SUCCESS; - // The numbers are: - // - size of plugin id param length (16 bits), holding the value 4 - // - size of event size param (16 bits), holding the event length - // - plugin id (32 bits) - uint32_t reqsize = sizeof(scap_evt) + 2 + 2 + 4 + plugin_evt->datalen; + /* + * | scap_evt | len_id (4B) | len_pl (4B) | id | payload | + * Note: we need to use 4B for len_id too because the PPME_PLUGINEVENT_E has + * EF_LARGE_PAYLOAD flag! + */ + uint32_t reqsize = sizeof(scap_evt) + 4 + 4 + 4 + plugin_evt->datalen; if(handle->m_input_plugin_evt_storage_len < reqsize) { - handle->m_input_plugin_evt_storage = (uint8_t*)realloc(handle->m_input_plugin_evt_storage, reqsize); - handle->m_input_plugin_evt_storage_len = reqsize; + uint8_t *tmp = (uint8_t*)realloc(handle->m_input_plugin_evt_storage, reqsize); + if (tmp) + { + handle->m_input_plugin_evt_storage = tmp; + handle->m_input_plugin_evt_storage_len = reqsize; + } + else + { + snprintf(handle->m_lasterr, SCAP_LASTERR_SIZE, "%s", "failed to alloc space for plugin storage"); + ASSERT(false); + return SCAP_FAILURE; + } } scap_evt* evt = (scap_evt*)handle->m_input_plugin_evt_storage; @@ -1793,12 +1805,11 @@ static int32_t scap_next_plugin(scap_t* handle, OUT scap_evt** pevent, OUT uint1 uint8_t* buf = handle->m_input_plugin_evt_storage + sizeof(scap_evt); - const uint16_t plugin_id_size = 4; + const uint32_t plugin_id_size = 4; memcpy(buf, &plugin_id_size, sizeof(plugin_id_size)); buf += sizeof(plugin_id_size); - // Plugin event sizes are 32 bits but param sizes are 16 bits - uint16_t datalen = plugin_evt->datalen; + uint32_t datalen = plugin_evt->datalen; memcpy(buf, &(datalen), sizeof(datalen)); buf += sizeof(datalen); diff --git a/userspace/libscap/scap.h b/userspace/libscap/scap.h index ed7626add..b6fb801e6 100644 --- a/userspace/libscap/scap.h +++ b/userspace/libscap/scap.h @@ -511,7 +511,8 @@ typedef enum scap_dump_flags SCAP_DF_NONE = 0, SCAP_DF_STATE_ONLY = 1, ///< The event should be used for state update but it should ///< not be shown to the user - SCAP_DF_TRACER = (1 << 1) ///< This event is a tracer + SCAP_DF_TRACER = (1 << 1), ///< This event is a tracer + SCAP_DF_LARGE = (1 << 2) ///< This event has large payload (up to UINT_MAX Bytes, ie 4GB) }scap_dump_flags; typedef struct scap_dumper scap_dumper_t; diff --git a/userspace/libscap/scap_savefile.c b/userspace/libscap/scap_savefile.c index c3505c331..aad9737c1 100755 --- a/userspace/libscap/scap_savefile.c +++ b/userspace/libscap/scap_savefile.c @@ -1151,13 +1151,15 @@ int32_t scap_dump(scap_t *handle, scap_dumper_t *d, scap_evt *e, uint16_t cpuid, { block_header bh; uint32_t bt; + bool large_payload = flags & SCAP_DF_LARGE; + flags &= ~SCAP_DF_LARGE; if(flags == 0) { // // Write the section header // - bh.block_type = EV_BLOCK_TYPE_V2; + bh.block_type = large_payload ? EV_BLOCK_TYPE_V2_LARGE : EV_BLOCK_TYPE_V2; bh.block_total_length = scap_normalize_block_len(sizeof(block_header) + sizeof(cpuid) + e->len + 4); bt = bh.block_total_length; @@ -1176,7 +1178,7 @@ int32_t scap_dump(scap_t *handle, scap_dumper_t *d, scap_evt *e, uint16_t cpuid, // // Write the section header // - bh.block_type = EVF_BLOCK_TYPE_V2; + bh.block_type = large_payload ? EVF_BLOCK_TYPE_V2_LARGE : EVF_BLOCK_TYPE_V2; bh.block_total_length = scap_normalize_block_len(sizeof(block_header) + sizeof(cpuid) + sizeof(flags) + e->len + 4); bt = bh.block_total_length; @@ -2650,6 +2652,8 @@ int32_t scap_read_init(scap_t *handle, gzFile f) case EV_BLOCK_TYPE_V2: case EVF_BLOCK_TYPE: case EVF_BLOCK_TYPE_V2: + case EV_BLOCK_TYPE_V2_LARGE: + case EVF_BLOCK_TYPE_V2_LARGE: found_ev = 1; // @@ -2802,9 +2806,11 @@ int32_t scap_next_offline(scap_t *handle, OUT scap_evt **pevent, OUT uint16_t *p if(bh.block_type != EV_BLOCK_TYPE && bh.block_type != EV_BLOCK_TYPE_V2 && + bh.block_type != EV_BLOCK_TYPE_V2_LARGE && bh.block_type != EV_BLOCK_TYPE_INT && bh.block_type != EVF_BLOCK_TYPE && - bh.block_type != EVF_BLOCK_TYPE_V2) + bh.block_type != EVF_BLOCK_TYPE_V2 && + bh.block_type != EVF_BLOCK_TYPE_V2_LARGE) { snprintf(handle->m_lasterr, SCAP_LASTERR_SIZE, "unexpected block type %u", (uint32_t)bh.block_type); handle->m_unexpected_block_readsize = readsize; @@ -2812,7 +2818,10 @@ int32_t scap_next_offline(scap_t *handle, OUT scap_evt **pevent, OUT uint16_t *p } hdr_len = sizeof(struct ppm_evt_hdr); - if(bh.block_type != EV_BLOCK_TYPE_V2 && bh.block_type != EVF_BLOCK_TYPE_V2) + if(bh.block_type != EV_BLOCK_TYPE_V2 && + bh.block_type != EV_BLOCK_TYPE_V2_LARGE && + bh.block_type != EVF_BLOCK_TYPE_V2 && + bh.block_type != EVF_BLOCK_TYPE_V2_LARGE) { hdr_len -= 4; } @@ -2827,11 +2836,25 @@ int32_t scap_next_offline(scap_t *handle, OUT scap_evt **pevent, OUT uint16_t *p // Read the event // readlen = bh.block_total_length - sizeof(bh); - if (readlen > FILE_READ_BUF_SIZE) { - snprintf(handle->m_lasterr, SCAP_LASTERR_SIZE, "event block length %u greater than read buffer size %u", - readlen, - FILE_READ_BUF_SIZE); - return SCAP_FAILURE; + // Non-large block types have an uint16_max maximum size + if (bh.block_type != EV_BLOCK_TYPE_V2_LARGE && bh.block_type != EVF_BLOCK_TYPE_V2_LARGE) { + if(readlen > FILE_READ_BUF_SIZE) { + snprintf(handle->m_lasterr, SCAP_LASTERR_SIZE, "event block length %u greater than NON-LARGE read buffer size %u", + readlen, + FILE_READ_BUF_SIZE); + return SCAP_FAILURE; + } + } else if (readlen > handle->m_file_evt_buf_size) { + // Try to allocate a buffer large enough + char *tmp = realloc(handle->m_file_evt_buf, readlen); + if (!tmp) { + snprintf(handle->m_lasterr, SCAP_LASTERR_SIZE, "event block length %u greater than read buffer size %zu", + readlen, + handle->m_file_evt_buf_size); + return SCAP_FAILURE; + } + handle->m_file_evt_buf = tmp; + handle->m_file_evt_buf_size = readlen; } readsize = gzread(f, handle->m_file_evt_buf, readlen); @@ -2842,7 +2865,7 @@ int32_t scap_next_offline(scap_t *handle, OUT scap_evt **pevent, OUT uint16_t *p // *pcpuid = *(uint16_t *)handle->m_file_evt_buf; - if(bh.block_type == EVF_BLOCK_TYPE || bh.block_type == EVF_BLOCK_TYPE_V2) + if(bh.block_type == EVF_BLOCK_TYPE || bh.block_type == EVF_BLOCK_TYPE_V2 || bh.block_type == EVF_BLOCK_TYPE_V2_LARGE) { handle->m_last_evt_dump_flags = *(uint32_t*)(handle->m_file_evt_buf + sizeof(uint16_t)); *pevent = (struct ppm_evt_hdr *)(handle->m_file_evt_buf + sizeof(uint16_t) + sizeof(uint32_t)); @@ -2862,10 +2885,13 @@ int32_t scap_next_offline(scap_t *handle, OUT scap_evt **pevent, OUT uint16_t *p continue; } - if(bh.block_type != EV_BLOCK_TYPE_V2 && bh.block_type != EVF_BLOCK_TYPE_V2) + if(bh.block_type != EV_BLOCK_TYPE_V2 && + bh.block_type != EV_BLOCK_TYPE_V2_LARGE && + bh.block_type != EVF_BLOCK_TYPE_V2 && + bh.block_type != EVF_BLOCK_TYPE_V2_LARGE) { // - // We're reading a old capture which events don't have nparams in the header. + // We're reading an old capture whose events don't have nparams in the header. // Convert it to the current version. // if((readlen + sizeof(uint32_t)) > FILE_READ_BUF_SIZE) diff --git a/userspace/libscap/scap_savefile.h b/userspace/libscap/scap_savefile.h index ebbcc16f3..0fee9daad 100644 --- a/userspace/libscap/scap_savefile.h +++ b/userspace/libscap/scap_savefile.h @@ -123,6 +123,8 @@ typedef struct _section_header_block // backward compatibility #define EV_BLOCK_TYPE_V2 0x216 +#define EV_BLOCK_TYPE_V2_LARGE 0x221 + /////////////////////////////////////////////////////////////////////////////// // INTERFACE LIST BLOCK /////////////////////////////////////////////////////////////////////////////// @@ -148,6 +150,8 @@ typedef struct _section_header_block #define EVF_BLOCK_TYPE_V2 0x217 +#define EVF_BLOCK_TYPE_V2_LARGE 0x222 + #if defined __sun #pragma pack() #else diff --git a/userspace/libsinsp/container.cpp b/userspace/libsinsp/container.cpp index f502c754a..ea6620544 100644 --- a/userspace/libsinsp/container.cpp +++ b/userspace/libsinsp/container.cpp @@ -245,7 +245,7 @@ string sinsp_container_manager::container_to_json(const sinsp_container_info& co bool sinsp_container_manager::container_to_sinsp_event(const string& json, sinsp_evt* evt, shared_ptr tinfo) { - size_t totlen = sizeof(scap_evt) + sizeof(uint16_t) + json.length() + 1; + size_t totlen = sizeof(scap_evt) + sizeof(uint32_t) + json.length() + 1; ASSERT(evt->m_pevt_storage == nullptr); evt->m_pevt_storage = new char[totlen]; @@ -273,10 +273,10 @@ bool sinsp_container_manager::container_to_sinsp_event(const string& json, sinsp scapevt->type = PPME_CONTAINER_JSON_E; scapevt->nparams = 1; - uint16_t* lens = (uint16_t*)((char *)scapevt + sizeof(struct ppm_evt_hdr)); - char* valptr = (char*)lens + sizeof(uint16_t); + uint32_t* lens = (uint32_t*)((char *)scapevt + sizeof(struct ppm_evt_hdr)); + char* valptr = (char*)lens + sizeof(uint32_t); - *lens = (uint16_t)json.length() + 1; + *lens = (uint32_t)json.length() + 1; memcpy(valptr, json.c_str(), *lens); evt->init(); @@ -345,7 +345,7 @@ void sinsp_container_manager::dump_containers(scap_dumper_t* dumper) sinsp_evt evt; if(container_to_sinsp_event(container_to_json(*it.second), &evt, it.second->get_tinfo(m_inspector))) { - int32_t res = scap_dump(m_inspector->m_h, dumper, evt.m_pevt, evt.m_cpuid, 0); + int32_t res = scap_dump(m_inspector->m_h, dumper, evt.m_pevt, evt.m_cpuid, SCAP_DF_LARGE); if(res != SCAP_SUCCESS) { throw sinsp_exception(scap_getlasterr(m_inspector->m_h)); diff --git a/userspace/libsinsp/event.cpp b/userspace/libsinsp/event.cpp index e412d6417..749c42cc0 100644 --- a/userspace/libsinsp/event.cpp +++ b/userspace/libsinsp/event.cpp @@ -742,7 +742,7 @@ Json::Value sinsp_evt::get_param_as_json(uint32_t id, OUT const char** resolved_ { const ppm_param_info* param_info; char* payload; - uint16_t payload_len; + uint32_t payload_len; Json::Value ret; // @@ -1457,7 +1457,7 @@ const char* sinsp_evt::get_param_as_str(uint32_t id, OUT const char** resolved_s const ppm_param_info* param_info; char* payload; uint32_t j; - uint16_t payload_len; + uint32_t payload_len; // // Make sure the params are actually loaded @@ -2630,6 +2630,11 @@ scap_dump_flags sinsp_evt::get_dump_flags(OUT bool* should_drop) dflags |= SCAP_DF_TRACER; } + if(get_info_flags() & EF_LARGE_PAYLOAD) + { + dflags |= SCAP_DF_LARGE; + } + return (scap_dump_flags)dflags; } #endif diff --git a/userspace/libsinsp/event.h b/userspace/libsinsp/event.h index ed2a6e2ea..3f38561d1 100644 --- a/userspace/libsinsp/event.h +++ b/userspace/libsinsp/event.h @@ -85,9 +85,9 @@ class SINSP_PUBLIC sinsp_evt_param { public: char* m_val; ///< Pointer to the event parameter data. - uint16_t m_len; ///< Length of the parameter pointed by m_val. + uint32_t m_len; ///< Length of the parameter pointed by m_val. private: - inline void init(char* valptr, uint16_t len) + inline void init(char* valptr, uint32_t len) { m_val = valptr; m_len = len; @@ -447,16 +447,37 @@ class SINSP_PUBLIC sinsp_evt : public gen_event // event table entry may contain new parameters. // Use the minimum between the two values. nparams = m_info->nparams < m_pevt->nparams ? m_info->nparams : m_pevt->nparams; - uint16_t *lens = (uint16_t *)((char *)m_pevt + sizeof(struct ppm_evt_hdr)); - // The offset in the block is instead always based on the capture value. - char *valptr = (char *)lens + m_pevt->nparams * sizeof(uint16_t); + + char *valptr; + union { + uint16_t* lens16; + uint32_t* lens32; + } lens; + + const bool large_payload = get_info_flags() & EF_LARGE_PAYLOAD; + + if (large_payload) { + lens.lens32 = (uint32_t *)((char *)m_pevt + sizeof(struct ppm_evt_hdr)); + // The offset in the block is instead always based on the capture value. + valptr = (char *)lens.lens32 + m_pevt->nparams * sizeof(uint32_t); + } else + { + lens.lens16 = (uint16_t*)((char*)m_pevt + sizeof(struct ppm_evt_hdr)); + // The offset in the block is instead always based on the capture value. + valptr = (char *)lens.lens16 + m_pevt->nparams * sizeof(uint16_t); + } m_params.clear(); for(j = 0; j < nparams; j++) { - par.init(valptr, lens[j]); + if (large_payload) { + par.init(valptr, lens.lens32[j]); + valptr += lens.lens32[j]; + } else { + par.init(valptr, lens.lens16[j]); + valptr += lens.lens16[j]; + } m_params.push_back(par); - valptr += lens[j]; } } std::string get_param_value_str(uint32_t id, bool resolved); From b9d2fccebf0feb94729c78bee0368c70d9c24819 Mon Sep 17 00:00:00 2001 From: Mark Stemm Date: Wed, 27 Oct 2021 18:10:32 -0700 Subject: [PATCH 056/148] Use const char* / uint8_t* in Plugins API Now that memory is always either owned by the plugin framework (init config/open params) or owned by the plugin (data payloads, extracted strings, demographic info), all char*/uint8_t* values involved in the api should be const. Signed-off-by: Mark Stemm --- userspace/libscap/plugin_info.h | 56 ++++++++++++++++----------------- userspace/libsinsp/plugin.cpp | 12 +++---- userspace/libsinsp/plugin.h | 26 +++++++-------- 3 files changed, 47 insertions(+), 47 deletions(-) diff --git a/userspace/libscap/plugin_info.h b/userspace/libscap/plugin_info.h index 7ed929a24..140362c11 100644 --- a/userspace/libscap/plugin_info.h +++ b/userspace/libscap/plugin_info.h @@ -87,7 +87,7 @@ typedef enum ss_plugin_rc typedef struct ss_plugin_event { uint64_t evtnum; - uint8_t *data; + const uint8_t *data; uint32_t datalen; uint64_t ts; } ss_plugin_event; @@ -117,12 +117,12 @@ typedef struct ss_plugin_event typedef struct ss_plugin_extract_field { uint32_t field_id; - const char *field; - const char *arg; + const char* field; + const char* arg; uint32_t ftype; bool field_present; - char *res_str; + const char* res_str; uint64_t res_u64; } ss_plugin_extract_field; @@ -185,7 +185,7 @@ typedef struct // of the API they run against, and the engine will take care of checking // and enforcing compatibility. // - char* (*get_required_api_version)(); + const char* (*get_required_api_version)(); // // Return the plugin type. // Required: yes @@ -207,7 +207,7 @@ typedef struct // by the engine and passed to the other plugin functions. // If rc is SS_PLUGIN_FAILURE, this function should return NULL. // - ss_plugin_t* (*init)(char* config, ss_plugin_rc* rc); + ss_plugin_t* (*init)(const char* config, ss_plugin_rc* rc); // // Destroy the plugin and, if plugin state was allocated, free it. // Required: yes @@ -223,7 +223,7 @@ typedef struct // string with more context for the error. The plugin manager // calls get_last_error() to access that string. // - char* (*get_last_error)(ss_plugin_t* s); + const char* (*get_last_error)(ss_plugin_t* s); // // Return the unique ID of the plugin. // Required: yes @@ -236,19 +236,19 @@ typedef struct // information about the plugin. // Required: yes // - char* (*get_name)(); + const char* (*get_name)(); // // Return the descriptions of the plugin, which will be printed when displaying // information about the plugin or its events. // Required: yes // - char* (*get_description)(); + const char* (*get_description)(); // // Return a string containing contact info (url, email, twitter, etc) for // the plugin authors. // Required: yes // - char* (*get_contact)(); + const char* (*get_contact)(); // // Return the version of this plugin itself // Required: yes @@ -260,7 +260,7 @@ typedef struct // generated events must match the major version of the plugin // used to read events. // - char* (*get_version)(); + const char* (*get_version)(); // // Return a string describing the events generated by this source plugin. // Required: yes @@ -268,7 +268,7 @@ typedef struct // "k8s_audit", etc. The source can be used by extractor // plugins to filter the events they receive. // - char* (*get_event_source)(); + const char* (*get_event_source)(); // // Return the list of extractor fields exported by this plugin. Extractor // fields can be used in Falco rule conditions and sysdig filters. @@ -289,7 +289,7 @@ typedef struct // {"type": "string", "name": "field1", "argRequired": true, "desc": "Describing field 1"}, // {"type": "uint64", "name": "field2", "desc": "Describing field 2"} // ] - char* (*get_fields)(); + const char* (*get_fields)(); // // Open the source and start a capture (e.g. stream of events) // Required: yes @@ -302,7 +302,7 @@ typedef struct // Return value: a pointer to the open context that will be passed to next_batch(), // close(), event_to_string() and extract_fields. // - ss_instance_t* (*open)(ss_plugin_t* s, char* params, ss_plugin_rc* rc); + ss_instance_t* (*open)(ss_plugin_t* s, const char* params, ss_plugin_rc* rc); // // Close a capture. // Required: yes @@ -330,7 +330,7 @@ typedef struct // when possible, it's recommended as it provides valuable information to the // user. // - char* (*get_progress)(ss_plugin_t* s, ss_instance_t* h, uint32_t* progress_pct); + const char* (*get_progress)(ss_plugin_t* s, ss_instance_t* h, uint32_t* progress_pct); // // Return a text representation of an event generated by this source plugin. // Required: yes @@ -343,7 +343,7 @@ typedef struct // and must not be deallocated or modified until the next call to // event_to_string(). // - char *(*event_to_string)(ss_plugin_t *s, const uint8_t *data, uint32_t datalen); + const char* (*event_to_string)(ss_plugin_t *s, const uint8_t *data, uint32_t datalen); // // Extract one or more a filter field values from an event. // Required: no @@ -358,7 +358,7 @@ typedef struct // extracted value as output. Memory pointers set as output must be allocated // by the plugin and must not be deallocated or modified until the next // extract_fields() call. - // + // // Return value: An ss_plugin_rc with values SS_PLUGIN_SUCCESS or SS_PLUGIN_FAILURE. // ss_plugin_rc (*extract_fields)(ss_plugin_t *s, const ss_plugin_event *evt, uint32_t num_fields, ss_plugin_extract_field *fields); @@ -381,7 +381,7 @@ typedef struct ss_plugin_t* state; ss_instance_t* handle; uint32_t id; - char *name; + const char* name; } source_plugin_info; // @@ -399,7 +399,7 @@ typedef struct // of the API they run against, and the engine will take care of checking // and enforcing compatibility. // - char* (*get_required_api_version)(); + const char* (*get_required_api_version)(); // // Return the plugin type. // Required: yes @@ -420,7 +420,7 @@ typedef struct // Return value: pointer to the plugin state that will be treated as opaque // by the engine and passed to the other plugin functions. // - ss_plugin_t* (*init)(char* config, int32_t* rc); + ss_plugin_t* (*init)(const char* config, int32_t* rc); // // Destroy the plugin and, if plugin state was allocated, free it. // Required: yes @@ -436,25 +436,25 @@ typedef struct // string with more context for the error. The plugin manager // calls get_last_error() to access that string. // - char* (*get_last_error)(ss_plugin_t* s); + const char* (*get_last_error)(ss_plugin_t* s); // // Return the name of the plugin, which will be printed when displaying // information about the plugin. // Required: yes // - char* (*get_name)(); + const char* (*get_name)(); // // Return the descriptions of the plugin, which will be printed when displaying // information about the plugin or its events. // Required: yes // - char* (*get_description)(); + const char* (*get_description)(); // // Return a string containing contact info (url, email, twitter, etc) for // the plugin author. // Required: yes // - char* (*get_contact)(); + const char* (*get_contact)(); // // Return the version of this plugin itself // Required: yes @@ -466,7 +466,7 @@ typedef struct // generated events must match the major version of the plugin // used to read events. // - char* (*get_version)(); + const char* (*get_version)(); // // Return a string describing the event sources that this // extractor plugin can consume. @@ -477,7 +477,7 @@ typedef struct // This function is optional--if NULL then the exctractor // plugin will receive every event. // - char* (*get_extract_event_sources)(); + const char* (*get_extract_event_sources)(); // // Return the list of extractor fields exported by this plugin. Extractor // fields can be used in Falco rules and sysdig filters. @@ -485,7 +485,7 @@ typedef struct // Return value: a string with the list of fields encoded as a json // array. // - char* (*get_fields)(); + const char* (*get_fields)(); // // Extract one or more a filter field values from an event. // Required: yes @@ -497,7 +497,7 @@ typedef struct // extracted value as output. Memory pointers set as output must be allocated // by the plugin and must not be deallocated or modified until the next // extract_fields() call. - // + // // Return value: An integer with values SS_PLUGIN_SUCCESS or SS_PLUGIN_FAILURE. // ss_plugin_rc (*extract_fields)(ss_plugin_t *s, const ss_plugin_event *evt, uint32_t num_fields, ss_plugin_extract_field *fields); diff --git a/userspace/libsinsp/plugin.cpp b/userspace/libsinsp/plugin.cpp index d228d571a..5d9e14711 100755 --- a/userspace/libsinsp/plugin.cpp +++ b/userspace/libsinsp/plugin.cpp @@ -365,7 +365,7 @@ std::string sinsp_plugin::version::as_string() const std::shared_ptr sinsp_plugin::register_plugin(sinsp* inspector, string filepath, - char* config, + const char* config, filter_check_list &available_checks) { string errstr; @@ -400,7 +400,7 @@ std::shared_ptr sinsp_plugin::register_plugin(sinsp* inspector, return plugin; } -std::shared_ptr sinsp_plugin::create_plugin(string &filepath, char* config, std::string &errstr) +std::shared_ptr sinsp_plugin::create_plugin(string &filepath, const char* config, std::string &errstr) { std::shared_ptr ret; @@ -525,7 +525,7 @@ sinsp_plugin::~sinsp_plugin() { } -bool sinsp_plugin::init(char *config) +bool sinsp_plugin::init(const char *config) { if (!m_plugin_info.init) { @@ -669,7 +669,7 @@ void* sinsp_plugin::getsym(void* handle, const char* name, std::string &errstr) } // Used below--set a std::string from the provided allocated charbuf and free() the charbuf. -std::string sinsp_plugin::str_from_alloc_charbuf(char *charbuf) +std::string sinsp_plugin::str_from_alloc_charbuf(const char* charbuf) { std::string str; @@ -722,7 +722,7 @@ bool sinsp_plugin::resolve_dylib_symbols(void *handle, std::string &errstr) // if(m_plugin_info.get_fields) { - char* sfields = m_plugin_info.get_fields(); + const char* sfields = m_plugin_info.get_fields(); if(sfields == NULL) { throw sinsp_exception(string("error in plugin ") + m_name + ": get_fields returned a null string"); @@ -839,7 +839,7 @@ source_plugin_info *sinsp_source_plugin::plugin_info() return &m_source_plugin_info; } -bool sinsp_source_plugin::open(char *params, ss_plugin_rc &rc) +bool sinsp_source_plugin::open(const char *params, ss_plugin_rc &rc) { ss_plugin_rc orc; diff --git a/userspace/libsinsp/plugin.h b/userspace/libsinsp/plugin.h index 2bbdb0f62..bcdeb1d18 100755 --- a/userspace/libsinsp/plugin.h +++ b/userspace/libsinsp/plugin.h @@ -82,13 +82,13 @@ class sinsp_plugin // The created sinsp_plugin is returned. static std::shared_ptr register_plugin(sinsp* inspector, std::string filepath, - char *config, + const char* config, filter_check_list &available_checks = g_filterlist); // Create a plugin from the dynamic library at the provided // path. On error, the shared_ptr will == NULL and errstr is // set with an error. - static std::shared_ptr create_plugin(std::string &filepath, char *config, std::string &errstr); + static std::shared_ptr create_plugin(std::string &filepath, const char* config, std::string &errstr); // Return a string with names/descriptions/etc of all plugins used by this inspector static std::list plugin_infos(sinsp *inspector); @@ -102,7 +102,7 @@ class sinsp_plugin // Returns true on success, false + sets errstr on error. virtual bool resolve_dylib_symbols(void *handle, std::string &errstr); - bool init(char *config); + bool init(const char* config); void destroy(); virtual ss_plugin_type type() = 0; @@ -124,7 +124,7 @@ class sinsp_plugin static void* getsym(void* handle, const char* name, std::string &errstr); // Helper function to set a string from an allocated charbuf and free the charbuf. - std::string str_from_alloc_charbuf(char *charbuf); + std::string str_from_alloc_charbuf(const char* charbuf); // init() will call this to save the resulting state struct virtual void set_plugin_state(ss_plugin_t *state) = 0; @@ -135,15 +135,15 @@ class sinsp_plugin // types. get_required_api_version/get_type are common but not // included here as they are called in create_plugin() typedef struct { - char* (*get_required_api_version)(); - ss_plugin_t* (*init)(char* config, ss_plugin_rc* rc); + const char* (*get_required_api_version)(); + ss_plugin_t* (*init)(const char* config, ss_plugin_rc* rc); void (*destroy)(ss_plugin_t* s); - char* (*get_last_error)(ss_plugin_t* s); - char* (*get_name)(); - char* (*get_description)(); - char* (*get_contact)(); - char* (*get_version)(); - char* (*get_fields)(); + const char* (*get_last_error)(ss_plugin_t* s); + const char* (*get_name)(); + const char* (*get_description)(); + const char* (*get_contact)(); + const char* (*get_version)(); + const char* (*get_fields)(); ss_plugin_rc (*extract_fields)(ss_plugin_t *s, const ss_plugin_event *evt, uint32_t num_fields, ss_plugin_extract_field *fields); } common_plugin_info; @@ -179,7 +179,7 @@ class sinsp_source_plugin : public sinsp_plugin // Note that embedding ss_instance_t in the object means that // a plugin can only have one open active at a time. - bool open(char *params, ss_plugin_rc &rc); + bool open(const char* params, ss_plugin_rc &rc); void close(); std::string get_progress(uint32_t &progress_pct); From 6eb1a755a92f690d9a63302f3c3a3c3fae95a3e1 Mon Sep 17 00:00:00 2001 From: Federico Di Pierro Date: Wed, 27 Oct 2021 14:42:05 +0200 Subject: [PATCH 057/148] fix(driver/ebpf): fixed eBPF issues on clang5: * bound check state->n_drops_X counters * moved off_bounded declaration near its usage in bpf_parse_readv_writev_bufs() * force-disable switch jump table Signed-off-by: Federico Di Pierro --- driver/bpf/fillers.h | 15 ++++++++++----- driver/ppm_flag_helpers.h | 7 +++++++ 2 files changed, 17 insertions(+), 5 deletions(-) diff --git a/driver/bpf/fillers.h b/driver/bpf/fillers.h index 2326bf28f..a4377651b 100644 --- a/driver/bpf/fillers.h +++ b/driver/bpf/fillers.h @@ -100,19 +100,25 @@ FILLER_RAW(terminate_filler) bpf_printk("PPM_FAILURE_BUFFER_FULL event=%d curarg=%d\n", state->tail_ctx.evt_type, state->tail_ctx.curarg); - ++state->n_drops_buffer; + if (state->n_drops_buffer != ULLONG_MAX) { + ++state->n_drops_buffer; + } break; case PPM_FAILURE_INVALID_USER_MEMORY: bpf_printk("PPM_FAILURE_INVALID_USER_MEMORY event=%d curarg=%d\n", state->tail_ctx.evt_type, state->tail_ctx.curarg); - ++state->n_drops_pf; + if (state->n_drops_pf != ULLONG_MAX) { + ++state->n_drops_pf; + } break; case PPM_FAILURE_BUG: bpf_printk("PPM_FAILURE_BUG event=%d curarg=%d\n", state->tail_ctx.evt_type, state->tail_ctx.curarg); - ++state->n_drops_bug; + if (state->n_drops_bug != ULLONG_MAX) { + ++state->n_drops_bug; + } break; case PPM_SKIP_EVENT: break; @@ -435,7 +441,6 @@ static __always_inline int bpf_parse_readv_writev_bufs(struct filler_data *data, if (flags & PRB_FLAG_PUSH_DATA) { if (size > 0) { unsigned long off = _READ(data->state->tail_ctx.curoff); - unsigned long off_bounded; unsigned long remaining = size; int j; @@ -446,7 +451,7 @@ static __always_inline int bpf_parse_readv_writev_bufs(struct filler_data *data, if (j == iovcnt) break; - off_bounded = off & SCRATCH_SIZE_HALF; + unsigned long off_bounded = off & SCRATCH_SIZE_HALF; if (off > SCRATCH_SIZE_HALF) break; diff --git a/driver/ppm_flag_helpers.h b/driver/ppm_flag_helpers.h index ab4869fe8..8914b6ec2 100644 --- a/driver/ppm_flag_helpers.h +++ b/driver/ppm_flag_helpers.h @@ -831,6 +831,12 @@ static __always_inline u8 sockopt_optname_to_scap(int level, int optname) case SO_COOKIE: return PPM_SOCKOPT_SO_COOKIE; #endif + case INT_MAX: + // forcefully disable switch jump table (clang-5 bug?) + // DO NOT merge with below default case + // otherwise this label will be skipped by compiler + ASSERT(false); + return PPM_SOCKOPT_UNKNOWN; default: ASSERT(false); return PPM_SOCKOPT_UNKNOWN; @@ -1212,6 +1218,7 @@ static __always_inline u32 semctl_cmd_to_scap(unsigned cmd) case GETZCNT: return PPM_GETZCNT; case SETALL: return PPM_SETALL; case SETVAL: return PPM_SETVAL; + case INT_MAX: return 0; // forcefully disable switch jump table (clang-5 bug?) } return 0; } From 39145281addcfa911191a2750f426bdc9c394523 Mon Sep 17 00:00:00 2001 From: Federico Di Pierro Date: Thu, 28 Oct 2021 11:35:14 +0200 Subject: [PATCH 058/148] update(driver/ebpf): added better explanation of switch workaround. Signed-off-by: Federico Di Pierro --- driver/ppm_flag_helpers.h | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/driver/ppm_flag_helpers.h b/driver/ppm_flag_helpers.h index 8914b6ec2..bcb4900d8 100644 --- a/driver/ppm_flag_helpers.h +++ b/driver/ppm_flag_helpers.h @@ -833,8 +833,14 @@ static __always_inline u8 sockopt_optname_to_scap(int level, int optname) #endif case INT_MAX: // forcefully disable switch jump table (clang-5 bug?) + // Basically, when labels values are similar AND the switch has many labels, + // compiler tends to build a jump table as optimization. + // This breaks with eBPF, and in our Makefile we already have the -fno-jump-tables; + // most probably clang5 had some kind of bug that caused -O2 mode to still use jump tables. + // Let's add a "very distant" label value to forcefully disable jump table. + // // DO NOT merge with below default case - // otherwise this label will be skipped by compiler + // otherwise this label will be skipped by compiler. ASSERT(false); return PPM_SOCKOPT_UNKNOWN; default: @@ -1218,7 +1224,8 @@ static __always_inline u32 semctl_cmd_to_scap(unsigned cmd) case GETZCNT: return PPM_GETZCNT; case SETALL: return PPM_SETALL; case SETVAL: return PPM_SETVAL; - case INT_MAX: return 0; // forcefully disable switch jump table (clang-5 bug?) + // forcefully disable switch jump table, see sockopt_optname_to_scap() for more info + case INT_MAX: return 0; } return 0; } From e879601cb3469d190be189113adc03645e274c81 Mon Sep 17 00:00:00 2001 From: Federico Di Pierro Date: Thu, 28 Oct 2021 15:27:57 +0200 Subject: [PATCH 059/148] update(driver/ebpf): only build workaround label case for BPF tracing, not kmod: kmod allows jump table usage. Signed-off-by: Federico Di Pierro --- driver/ppm_flag_helpers.h | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/driver/ppm_flag_helpers.h b/driver/ppm_flag_helpers.h index bcb4900d8..03ef765b0 100644 --- a/driver/ppm_flag_helpers.h +++ b/driver/ppm_flag_helpers.h @@ -831,6 +831,7 @@ static __always_inline u8 sockopt_optname_to_scap(int level, int optname) case SO_COOKIE: return PPM_SOCKOPT_SO_COOKIE; #endif +#ifdef __BPF_TRACING__ case INT_MAX: // forcefully disable switch jump table (clang-5 bug?) // Basically, when labels values are similar AND the switch has many labels, @@ -843,6 +844,7 @@ static __always_inline u8 sockopt_optname_to_scap(int level, int optname) // otherwise this label will be skipped by compiler. ASSERT(false); return PPM_SOCKOPT_UNKNOWN; +#endif default: ASSERT(false); return PPM_SOCKOPT_UNKNOWN; @@ -1224,8 +1226,10 @@ static __always_inline u32 semctl_cmd_to_scap(unsigned cmd) case GETZCNT: return PPM_GETZCNT; case SETALL: return PPM_SETALL; case SETVAL: return PPM_SETVAL; +#ifdef __BPF_TRACING__ // forcefully disable switch jump table, see sockopt_optname_to_scap() for more info case INT_MAX: return 0; +#endif } return 0; } From f994f47b2702ad2cf045ec023b0216e4ffb150b9 Mon Sep 17 00:00:00 2001 From: Mark Stemm Date: Mon, 1 Nov 2021 14:50:12 -0700 Subject: [PATCH 060/148] Fix const warnings in libscap Now that strings are being returned as const char*, make sure the variable holding the returned value is const char*. Signed-off-by: Mark Stemm --- userspace/libscap/scap.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/userspace/libscap/scap.c b/userspace/libscap/scap.c index 156f61219..434ac8e4d 100644 --- a/userspace/libscap/scap.c +++ b/userspace/libscap/scap.c @@ -1018,7 +1018,7 @@ scap_t* scap_open_plugin_int(char *error, int32_t *rc, source_plugin_info* input if(*rc != SCAP_SUCCESS) { - char *errstr = handle->m_input_plugin->get_last_error(handle->m_input_plugin->state); + const char *errstr = handle->m_input_plugin->get_last_error(handle->m_input_plugin->state); snprintf(error, SCAP_LASTERR_SIZE, "%s", errstr); scap_close(handle); return NULL; @@ -1731,7 +1731,7 @@ static int32_t scap_next_plugin(scap_t* handle, OUT scap_evt** pevent, OUT uint1 { if(handle->m_input_plugin_last_batch_res != SCAP_TIMEOUT && handle->m_input_plugin_last_batch_res != SCAP_EOF) { - char *errstr = handle->m_input_plugin->get_last_error(handle->m_input_plugin->state); + const char *errstr = handle->m_input_plugin->get_last_error(handle->m_input_plugin->state); snprintf(handle->m_lasterr, SCAP_LASTERR_SIZE, "%s", errstr); } int32_t tres = handle->m_input_plugin_last_batch_res; @@ -1757,7 +1757,7 @@ static int32_t scap_next_plugin(scap_t* handle, OUT scap_evt** pevent, OUT uint1 { if(handle->m_input_plugin_last_batch_res != SCAP_TIMEOUT && handle->m_input_plugin_last_batch_res != SCAP_EOF) { - char *errstr = handle->m_input_plugin->get_last_error(handle->m_input_plugin->state); + const char *errstr = handle->m_input_plugin->get_last_error(handle->m_input_plugin->state); snprintf(handle->m_lasterr, SCAP_LASTERR_SIZE, "%s", errstr); } return handle->m_input_plugin_last_batch_res; From 078d395940b0ab91f8e2090ceac595779f53ccc4 Mon Sep 17 00:00:00 2001 From: Mark Stemm Date: Wed, 27 Oct 2021 15:54:58 -0700 Subject: [PATCH 061/148] Don't define strlcpy when already defined (musl libc) Falco has a build variant that uses musl libc, and musl libc already defines strlcpy, so we don't want strlcpy when compiling with musl libc. So check for it at cmake time and if found set HAVE_STRLCPY. And only include the one in strlcpy.h if HAVE_STRLCPY is not defined. Signed-off-by: Mark Stemm --- CMakeLists.txt | 6 ++++++ userspace/common/strlcpy.h | 4 ++++ 2 files changed, 10 insertions(+) diff --git a/CMakeLists.txt b/CMakeLists.txt index 2b7544dba..1d70f0c7a 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -69,3 +69,9 @@ if(CREATE_TEST_TARGETS AND NOT WIN32) COMMAND ${CMAKE_MAKE_PROGRAM} run-unit-test-libsinsp ) endif() + +include(CheckSymbolExists) +check_symbol_exists(strlcpy "string.h" HAVE_STRLCPY) +if(HAVE_STRLCPY) + add_definitions(-DHAVE_STRLCPY) +endif() diff --git a/userspace/common/strlcpy.h b/userspace/common/strlcpy.h index 7a9f44360..8a15d4051 100644 --- a/userspace/common/strlcpy.h +++ b/userspace/common/strlcpy.h @@ -17,12 +17,15 @@ limitations under the License. #include #include +#pragma once /*! \brief Copy up to size - 1 characters from the NUL-terminated string src to dst, NUL-terminating the result. \return The length of the source string. */ + +#ifndef HAVE_STRLCPY static inline size_t strlcpy(char *dst, const char *src, size_t size) { size_t srcsize = strlen(src); if (size == 0) { @@ -40,3 +43,4 @@ static inline size_t strlcpy(char *dst, const char *src, size_t size) { return srcsize; } +#endif From cc44a9013460e6d20322a69d08ba5330591b5bb9 Mon Sep 17 00:00:00 2001 From: Jason Dellaluce Date: Tue, 2 Nov 2021 16:00:46 +0000 Subject: [PATCH 062/148] fix: check that field is present before copying it This caused spurious segmentation faults, because generally there is no guarantee that the plugin sets res_str to NULL if field_present is set to false. Co-authored-by: Leonardo Grasso Signed-off-by: Jason Dellaluce --- userspace/libsinsp/plugin.cpp | 26 ++++++++++++++------------ 1 file changed, 14 insertions(+), 12 deletions(-) diff --git a/userspace/libsinsp/plugin.cpp b/userspace/libsinsp/plugin.cpp index 5d9e14711..9d001e69f 100755 --- a/userspace/libsinsp/plugin.cpp +++ b/userspace/libsinsp/plugin.cpp @@ -631,18 +631,20 @@ bool sinsp_plugin::extract_field(ss_plugin_event &evt, sinsp_plugin::ext_field & } field.field_present = efield.field_present; - switch(field.ftype) - { - case PT_CHARBUF: - field.res_str = str_from_alloc_charbuf(efield.res_str); - break; - case PT_UINT64: - field.res_u64 = efield.res_u64; - break; - default: - ASSERT(false); - throw sinsp_exception("plugin extract error: unsupported field type " + to_string(field.ftype)); - break; + if (field.field_present) { + switch(field.ftype) + { + case PT_CHARBUF: + field.res_str = str_from_alloc_charbuf(efield.res_str); + break; + case PT_UINT64: + field.res_u64 = efield.res_u64; + break; + default: + ASSERT(false); + throw sinsp_exception("plugin extract error: unsupported field type " + to_string(field.ftype)); + break; + } } return true; From c4eab4e5d0d4e32df34b847bd8f7e0d71084cdae Mon Sep 17 00:00:00 2001 From: Radu Andries Date: Fri, 29 Oct 2021 15:44:22 +0200 Subject: [PATCH 063/148] Do not overwrite imageID if it's already set In environments using CRI imageID can be recovered in 2 ways, by reading it from the container info or by doing an extra query. Due to the extra query not working in gke environments, it's safer to check whether imageID has already been found instead of unconditionally overwrite it. Signed-off-by: Radu Andries --- userspace/libsinsp/container_engine/cri.cpp | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/userspace/libsinsp/container_engine/cri.cpp b/userspace/libsinsp/container_engine/cri.cpp index 22fcb5f3d..35883dd0d 100644 --- a/userspace/libsinsp/container_engine/cri.cpp +++ b/userspace/libsinsp/container_engine/cri.cpp @@ -145,7 +145,10 @@ bool cri_async_source::parse_cri(sinsp_container_info& container, const libsinsp { container.m_container_ip = m_cri->get_container_ip(container.m_id); } - container.m_imageid = m_cri->get_container_image_id(resp_container.image_ref()); + if(container.m_imageid.empty()) + { + container.m_imageid = m_cri->get_container_image_id(resp_container.image_ref()); + } } return true; From 1d9e61ebf6773367f12cd5d6b66e788d93ae545c Mon Sep 17 00:00:00 2001 From: Kevin Kauffman Date: Tue, 12 Oct 2021 17:49:43 -0400 Subject: [PATCH 064/148] expose API to set track connection status on parser Signed-off-by: Kevin Kauffman --- userspace/libsinsp/parsers.cpp | 5 +++++ userspace/libsinsp/parsers.h | 1 + 2 files changed, 6 insertions(+) diff --git a/userspace/libsinsp/parsers.cpp b/userspace/libsinsp/parsers.cpp index 6f10334eb..0e1195abe 100644 --- a/userspace/libsinsp/parsers.cpp +++ b/userspace/libsinsp/parsers.cpp @@ -109,6 +109,11 @@ void sinsp_parser::init_metaevt(metaevents_state& evt_state, uint16_t evt_type, evt_state.m_metaevt.m_fdinfo = NULL; } +void sinsp_parser::set_track_connection_status(bool enabled) +{ + m_track_connection_status = enabled; +} + /////////////////////////////////////////////////////////////////////////////// // PROCESSING ENTRY POINT /////////////////////////////////////////////////////////////////////////////// diff --git a/userspace/libsinsp/parsers.h b/userspace/libsinsp/parsers.h index 9779e54b6..4caa60dc9 100644 --- a/userspace/libsinsp/parsers.h +++ b/userspace/libsinsp/parsers.h @@ -79,6 +79,7 @@ class sinsp_parser // static void init_scapevt(metaevents_state& evt_state, uint16_t evt_type, uint16_t buf_size); + void set_track_connection_status(bool enabled); private: // // Initializers From 6210e63799e434bb257e85405bd0dd4b088336b7 Mon Sep 17 00:00:00 2001 From: Luca Guerra Date: Tue, 2 Nov 2021 16:02:11 -0700 Subject: [PATCH 065/148] Set compiler flags before including the libs dirs Move the check for strlcpy to before including the libs dirs so CFLAGS is properly set. Also add some messages. Signed-off-by: Luca Guerra Co-authored-by: Mark Stemm --- CMakeLists.txt | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-) diff --git a/CMakeLists.txt b/CMakeLists.txt index 1d70f0c7a..ad8e0a428 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -52,6 +52,15 @@ endif() set(PACKAGE_NAME "sysdig") +include(CheckSymbolExists) +check_symbol_exists(strlcpy "string.h" HAVE_STRLCPY) +if(HAVE_STRLCPY) + message(STATUS "Existing strlcpy found, will *not* use local definition by setting -DHAVE_STRLCPY.") + add_definitions(-DHAVE_STRLCPY) +else() + message(STATUS "No strlcpy found, will use local definition") +endif() + include(CompilerFlags) option(WITH_CHISEL "Include chisel implementation" OFF) @@ -70,8 +79,3 @@ if(CREATE_TEST_TARGETS AND NOT WIN32) ) endif() -include(CheckSymbolExists) -check_symbol_exists(strlcpy "string.h" HAVE_STRLCPY) -if(HAVE_STRLCPY) - add_definitions(-DHAVE_STRLCPY) -endif() From f7877aaf46fadae787fc852a042c58ae1fafd517 Mon Sep 17 00:00:00 2001 From: Federico Di Pierro Date: Wed, 3 Nov 2021 10:02:46 +0100 Subject: [PATCH 066/148] fix(driver,userspace/libsinsp): use new PPME_CONTAINER_JSON_2_ events with large payload; leave old ones untouched to avoid breaking backward compatibility. This way, new scap files with PPME_CONTAINER_JSON_2_ events cannot be open by old falco; moreover, new falco can correctly open old PPME_CONTAINER_JSON_ events. Signed-off-by: Federico Di Pierro Co-authored-by: Mark Stemm --- driver/event_table.c | 6 ++++-- driver/ppm_events_public.h | 4 +++- userspace/libsinsp/container.cpp | 2 +- userspace/libsinsp/filterchecks.cpp | 2 +- userspace/libsinsp/parsers.cpp | 5 +++-- userspace/libsinsp/sinsp.cpp | 4 ++-- 6 files changed, 14 insertions(+), 9 deletions(-) diff --git a/driver/event_table.c b/driver/event_table.c index d4345ea51..0962cd153 100644 --- a/driver/event_table.c +++ b/driver/event_table.c @@ -284,7 +284,7 @@ const struct ppm_event_info g_event_info[PPM_EVENT_MAX] = { /* PPME_TRACER_X */{ "tracer", EC_OTHER, EF_NONE, 3, { { "id", PT_INT64, PF_DEC }, { "tags", PT_CHARBUFARRAY, PF_NA }, { "args", PT_CHARBUF_PAIR_ARRAY, PF_NA } } }, /* PPME_MESOS_E */{"mesos", EC_INTERNAL, EF_SKIPPARSERESET | EF_MODIFIES_STATE, 1, {{"json", PT_CHARBUF, PF_NA} } }, /* PPME_MESOS_X */{"NA4", EC_SYSTEM, EF_UNUSED, 0}, - /* PPME_CONTAINER_JSON_E */{"container", EC_PROCESS, EF_MODIFIES_STATE | EF_LARGE_PAYLOAD, 1, {{"json", PT_CHARBUF, PF_NA} } }, + /* PPME_CONTAINER_JSON_E */{"container", EC_PROCESS, EF_MODIFIES_STATE, 1, {{"json", PT_CHARBUF, PF_NA} } }, /* PPME_CONTAINER_JSON_X */{"container", EC_PROCESS, EF_UNUSED, 0}, /* PPME_SYSCALL_SETSID_E */{"setsid", EC_PROCESS, EF_MODIFIES_STATE, 0}, /* PPME_SYSCALL_SETSID_X */{"setsid", EC_PROCESS, EF_MODIFIES_STATE, 1, {{"res", PT_PID, PF_DEC} } }, @@ -335,7 +335,9 @@ const struct ppm_event_info g_event_info[PPM_EVENT_MAX] = { /* PPME_SYSCALL_USERFAULTFD_E */{"userfaultfd", EC_FILE, EF_CREATES_FD | EF_MODIFIES_STATE, 0}, /* PPME_SYSCALL_USERFAULTFD_X */{"userfaultfd", EC_FILE, EF_CREATES_FD | EF_MODIFIES_STATE, 2, {{"res", PT_ERRNO, PF_DEC}, {"flags", PT_FLAGS32, PF_HEX, file_flags} } }, /* PPME_PLUGINEVENT_E */{"pluginevent", EC_OTHER, EF_LARGE_PAYLOAD, 2, {{"plugin ID", PT_UINT32, PF_DEC}, {"event_data", PT_BYTEBUF, PF_NA} } }, - /* PPME_NA1 */{"pluginevent", EC_OTHER, EF_UNUSED, 0} + /* PPME_NA1 */{"pluginevent", EC_OTHER, EF_UNUSED, 0}, + /* PPME_CONTAINER_JSON_2_E */{"container", EC_PROCESS, EF_MODIFIES_STATE | EF_LARGE_PAYLOAD, 1, {{"json", PT_CHARBUF, PF_NA} } }, + /* PPME_CONTAINER_JSON_2_X */{"container", EC_PROCESS, EF_UNUSED, 0}, /* NB: Starting from scap version 1.2, event types will no longer be changed when an event is modified, and the only kind of change permitted for pre-existent events is adding parameters. * New event types are allowed only for new syscalls or new internal events. * The number of parameters can be used to differentiate between event versions. diff --git a/driver/ppm_events_public.h b/driver/ppm_events_public.h index 4924442ee..56bd2871e 100644 --- a/driver/ppm_events_public.h +++ b/driver/ppm_events_public.h @@ -963,7 +963,9 @@ enum ppm_event_type { PPME_SYSCALL_USERFAULTFD_X = 321, PPME_PLUGINEVENT_E = 322, PPME_PLUGINEVENT_X = 323, - PPM_EVENT_MAX = 324 + PPME_CONTAINER_JSON_2_E = 324, + PPME_CONTAINER_JSON_2_X = 325, + PPM_EVENT_MAX = 326 }; /*@}*/ diff --git a/userspace/libsinsp/container.cpp b/userspace/libsinsp/container.cpp index ea6620544..6481d4876 100644 --- a/userspace/libsinsp/container.cpp +++ b/userspace/libsinsp/container.cpp @@ -270,7 +270,7 @@ bool sinsp_container_manager::container_to_sinsp_event(const string& json, sinsp } scapevt->tid = -1; scapevt->len = (uint32_t)totlen; - scapevt->type = PPME_CONTAINER_JSON_E; + scapevt->type = PPME_CONTAINER_JSON_2_E; scapevt->nparams = 1; uint32_t* lens = (uint32_t*)((char *)scapevt + sizeof(struct ppm_evt_hdr)); diff --git a/userspace/libsinsp/filterchecks.cpp b/userspace/libsinsp/filterchecks.cpp index ccf476f25..23dec76b7 100644 --- a/userspace/libsinsp/filterchecks.cpp +++ b/userspace/libsinsp/filterchecks.cpp @@ -1238,7 +1238,7 @@ uint8_t* sinsp_filter_check_fd::extract(sinsp_evt *evt, OUT uint32_t* len, bool break; case TYPE_UID: { - if(evt->get_type() == PPME_CONTAINER_JSON_E) + if(evt->get_type() == PPME_CONTAINER_JSON_E || evt->get_type() == PPME_CONTAINER_JSON_2_E) { return NULL; } diff --git a/userspace/libsinsp/parsers.cpp b/userspace/libsinsp/parsers.cpp index 0e1195abe..00be8b04d 100644 --- a/userspace/libsinsp/parsers.cpp +++ b/userspace/libsinsp/parsers.cpp @@ -438,6 +438,7 @@ void sinsp_parser::process_event(sinsp_evt *evt) parse_container_evt(evt); // deprecated, only here for backwards compatibility break; case PPME_CONTAINER_JSON_E: + case PPME_CONTAINER_JSON_2_E: parse_container_json_evt(evt); break; case PPME_CPU_HOTPLUG_E: @@ -547,7 +548,7 @@ bool sinsp_parser::reset(sinsp_evt *evt) // cleared in init(). So only keep the threadinfo for "live" // containers. // - if (m_inspector->is_live() && etype == PPME_CONTAINER_JSON_E && evt->m_tinfo_ref != nullptr) + if (m_inspector->is_live() && (etype == PPME_CONTAINER_JSON_E || etype == PPME_CONTAINER_JSON_2_E) && evt->m_tinfo_ref != nullptr) { // this is a synthetic event generated by the container manager // the threadinfo should already be set properly @@ -609,7 +610,7 @@ bool sinsp_parser::reset(sinsp_evt *evt) query_os = true; } - if(etype == PPME_CONTAINER_JSON_E) + if(etype == PPME_CONTAINER_JSON_E || etype == PPME_CONTAINER_JSON_2_E) { evt->m_tinfo = nullptr; return true; diff --git a/userspace/libsinsp/sinsp.cpp b/userspace/libsinsp/sinsp.cpp index 58d460f53..a8285e97a 100644 --- a/userspace/libsinsp/sinsp.cpp +++ b/userspace/libsinsp/sinsp.cpp @@ -345,7 +345,7 @@ void sinsp::init() if(res == SCAP_SUCCESS) { - if((pevent->type != PPME_CONTAINER_E) && (pevent->type != PPME_CONTAINER_JSON_E)) + if((pevent->type != PPME_CONTAINER_E) && (pevent->type != PPME_CONTAINER_JSON_E) && (pevent->type != PPME_CONTAINER_JSON_2_E)) { break; } @@ -1196,7 +1196,7 @@ int32_t sinsp::next(OUT sinsp_evt **puevt) uint64_t ts = evt->get_ts(); - if(m_firstevent_ts == 0 && evt->m_pevt->type != PPME_CONTAINER_JSON_E) + if(m_firstevent_ts == 0 && evt->m_pevt->type != PPME_CONTAINER_JSON_E && evt->m_pevt->type != PPME_CONTAINER_JSON_2_E) { m_firstevent_ts = ts; } From 396f2c936a6941462c0ea7ff29fe6157481df96d Mon Sep 17 00:00:00 2001 From: Sachin Kumar Singh Date: Fri, 5 Nov 2021 17:38:07 +0530 Subject: [PATCH 067/148] Fix link for libraries contibutions in README.md Signed-off-by: Sachin Kumar Singh --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 1913f7f75..26000f4ce 100644 --- a/README.md +++ b/README.md @@ -1,3 +1,3 @@ # falcosecurity/libs -As per the [OSS Libraries Contribution Plan](https://github.com/falcosecurity/falco/blob/master/proposals/2021019-libraries-donation.md), this repository has been chosen to be the new home for **libsinsp**, **libscap**, the **kernel module driver** and the **eBPF driver sources**. +As per the [OSS Libraries Contribution Plan](https://github.com/falcosecurity/falco/blob/master/proposals/20210119-libraries-contribution.md), this repository has been chosen to be the new home for **libsinsp**, **libscap**, the **kernel module driver** and the **eBPF driver sources**. From d6da92557957d97d407ebb4396c71e509457739b Mon Sep 17 00:00:00 2001 From: Grzegorz Nosek Date: Fri, 16 Jul 2021 14:23:20 +0200 Subject: [PATCH 068/148] Rename docker_async_instruction to docker_lookup_request This should explain the purpose of this class better Signed-off-by: Grzegorz Nosek --- userspace/libsinsp/container_engine/docker.h | 16 ++-- .../container_engine/docker_common.cpp | 78 +++++++++---------- 2 files changed, 47 insertions(+), 47 deletions(-) diff --git a/userspace/libsinsp/container_engine/docker.h b/userspace/libsinsp/container_engine/docker.h index 2e6f1cd5a..af30815e4 100644 --- a/userspace/libsinsp/container_engine/docker.h +++ b/userspace/libsinsp/container_engine/docker.h @@ -47,19 +47,19 @@ class sinsp_threadinfo; namespace libsinsp { namespace container_engine { -struct docker_async_instruction +struct docker_lookup_request { - docker_async_instruction() : + docker_lookup_request() : request_rw_size(false) {} - docker_async_instruction(const std::string container_id_value, - bool rw_size_value) : + docker_lookup_request(const std::string container_id_value, + bool rw_size_value) : container_id(container_id_value), request_rw_size(rw_size_value) {} - bool operator<(const docker_async_instruction& rhs) const + bool operator<(const docker_lookup_request& rhs) const { if(container_id < rhs.container_id) { @@ -69,7 +69,7 @@ struct docker_async_instruction return request_rw_size < rhs.request_rw_size; } - bool operator==(const docker_async_instruction& rhs) const + bool operator==(const docker_lookup_request& rhs) const { return container_id == rhs.container_id && request_rw_size == rhs.request_rw_size; @@ -79,7 +79,7 @@ struct docker_async_instruction bool request_rw_size; }; -class docker_async_source : public sysdig::async_key_value_source +class docker_async_source : public sysdig::async_key_value_source { enum docker_response { @@ -108,7 +108,7 @@ class docker_async_source : public sysdig::async_key_value_sourcenotify_new_container(res); @@ -482,11 +482,11 @@ void docker::parse_docker_async(const string& container_id, container_cache_inte sinsp_container_info result; - docker_async_instruction instruction(container_id, false /*don't request size*/); - if(m_docker_info_source->lookup(instruction, result, cb)) + docker_lookup_request request(container_id, false /*don't request size*/); + if(m_docker_info_source->lookup(request, result, cb)) { // if a previous lookup call already found the metadata, process it now - cb(instruction, result); + cb(request, result); // This should *never* happen, as ttl is 0 (never wait) g_logger.format(sinsp_logger::SEV_ERROR, @@ -495,31 +495,31 @@ void docker::parse_docker_async(const string& container_id, container_cache_inte } } -bool docker_async_source::parse_docker(const docker_async_instruction& instruction, sinsp_container_info& container) +bool docker_async_source::parse_docker(const docker_lookup_request& request, sinsp_container_info& container) { string json; g_logger.format(sinsp_logger::SEV_DEBUG, "docker_async (%s): Looking up info for container", - instruction.container_id.c_str()); + request.container_id.c_str()); - std::string request = build_request("/containers/" + instruction.container_id + "/json"); - if(instruction.request_rw_size) + std::string api_request = build_request("/containers/" + request.container_id + "/json"); + if(request.request_rw_size) { - request += "?size=true"; + api_request += "?size=true"; } - docker_response resp = get_docker(request, json); + docker_response resp = get_docker(api_request, json); switch(resp) { case docker_response::RESP_BAD_REQUEST: g_logger.format(sinsp_logger::SEV_DEBUG, "docker_async (%s): Initial url fetch failed, trying w/o api version", - instruction.container_id.c_str()); + request.container_id.c_str()); m_api_version = ""; json = ""; - resp = get_docker(build_request("/containers/" + instruction.container_id + "/json"), json); + resp = get_docker(build_request("/containers/" + request.container_id + "/json"), json); if (resp == docker_response::RESP_OK) { break; @@ -528,7 +528,7 @@ bool docker_async_source::parse_docker(const docker_async_instruction& instructi case docker_response::RESP_ERROR: g_logger.format(sinsp_logger::SEV_DEBUG, "docker_async (%s): Url fetch failed, returning false", - instruction.container_id.c_str()); + request.container_id.c_str()); return false; @@ -538,7 +538,7 @@ bool docker_async_source::parse_docker(const docker_async_instruction& instructi g_logger.format(sinsp_logger::SEV_DEBUG, "docker_async (%s): Parsing containers response \"%s\"", - instruction.container_id.c_str(), + request.container_id.c_str(), json.c_str()); Json::Value root; @@ -548,7 +548,7 @@ bool docker_async_source::parse_docker(const docker_async_instruction& instructi { g_logger.format(sinsp_logger::SEV_ERROR, "docker_async (%s): Could not parse json \"%s\", returning false", - instruction.container_id.c_str(), + request.container_id.c_str(), json.c_str()); ASSERT(false); @@ -594,7 +594,7 @@ bool docker_async_source::parse_docker(const docker_async_instruction& instructi { g_logger.format(sinsp_logger::SEV_DEBUG, "docker_async (%s) image (%s): Fetching image info", - instruction.container_id.c_str(), + request.container_id.c_str(), container.m_imageid.c_str()); string img_json; @@ -608,7 +608,7 @@ bool docker_async_source::parse_docker(const docker_async_instruction& instructi { g_logger.format(sinsp_logger::SEV_DEBUG, "docker_async (%s) image (%s): Image info fetch returned \"%s\"", - instruction.container_id.c_str(), + request.container_id.c_str(), container.m_imageid.c_str(), img_json.c_str()); @@ -665,7 +665,7 @@ bool docker_async_source::parse_docker(const docker_async_instruction& instructi { g_logger.format(sinsp_logger::SEV_ERROR, "docker_async (%s) image (%s): Could not parse json image info \"%s\"", - instruction.container_id.c_str(), + request.container_id.c_str(), container.m_imageid.c_str(), img_json.c_str()); } @@ -674,7 +674,7 @@ bool docker_async_source::parse_docker(const docker_async_instruction& instructi { g_logger.format(sinsp_logger::SEV_ERROR, "docker_async (%s) image (%s): Could not fetch image info", - instruction.container_id.c_str(), + request.container_id.c_str(), container.m_imageid.c_str()); } @@ -721,16 +721,16 @@ bool docker_async_source::parse_docker(const docker_async_instruction& instructi // separate thread so this is ok. g_logger.format(sinsp_logger::SEV_DEBUG, "docker_async (%s), secondary (%s): Doing blocking fetch of secondary container", - instruction.container_id.c_str(), + request.container_id.c_str(), secondary_container_id.c_str()); - if(parse_docker(docker_async_instruction(secondary_container_id, - false /*don't request size since we just need the IP*/), + if(parse_docker(docker_lookup_request(secondary_container_id, + false /*don't request size since we just need the IP*/), pcnt)) { g_logger.format(sinsp_logger::SEV_DEBUG, "docker_async (%s), secondary (%s): Secondary fetch successful", - instruction.container_id.c_str(), + request.container_id.c_str(), secondary_container_id.c_str()); container.m_container_ip = pcnt.m_container_ip; } @@ -738,7 +738,7 @@ bool docker_async_source::parse_docker(const docker_async_instruction& instructi { g_logger.format(sinsp_logger::SEV_ERROR, "docker_async (%s), secondary (%s): Secondary fetch failed", - instruction.container_id.c_str(), + request.container_id.c_str(), secondary_container_id.c_str()); } } @@ -865,16 +865,16 @@ bool docker_async_source::parse_docker(const docker_async_instruction& instructi g_logger.format(sinsp_logger::SEV_DEBUG, "docker_async (%s): parse_docker returning true", - instruction.container_id.c_str()); + request.container_id.c_str()); return true; } void docker::update_with_size(const std::string &container_id) { - auto cb = [this](const docker_async_instruction& instruction, const sinsp_container_info& res) { + auto cb = [this](const docker_lookup_request& request, const sinsp_container_info& res) { g_logger.format(sinsp_logger::SEV_DEBUG, "docker_async (%s): with size callback result=%d", - instruction.container_id.c_str(), + request.container_id.c_str(), res.m_lookup_state); sinsp_container_info::ptr_t updated = make_shared(res); @@ -886,8 +886,8 @@ void docker::update_with_size(const std::string &container_id) container_id.c_str()); sinsp_container_info result; - docker_async_instruction instruction(container_id, true /*request rw size*/); - (void)m_docker_info_source->lookup(instruction, result, cb); + docker_lookup_request request(container_id, true /*request rw size*/); + (void)m_docker_info_source->lookup(request, result, cb); } #endif // _WIN32 From 52d12274ec57b542fd6950e00199ddb165912aea Mon Sep 17 00:00:00 2001 From: Grzegorz Nosek Date: Fri, 16 Jul 2021 14:25:05 +0200 Subject: [PATCH 069/148] Fix comparison of `docker_lookup_request`s Signed-off-by: Grzegorz Nosek --- userspace/libsinsp/container_engine/docker.h | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/userspace/libsinsp/container_engine/docker.h b/userspace/libsinsp/container_engine/docker.h index af30815e4..fbe472789 100644 --- a/userspace/libsinsp/container_engine/docker.h +++ b/userspace/libsinsp/container_engine/docker.h @@ -61,9 +61,9 @@ struct docker_lookup_request bool operator<(const docker_lookup_request& rhs) const { - if(container_id < rhs.container_id) + if(container_id != rhs.container_id) { - return true; + return container_id < rhs.container_id; } return request_rw_size < rhs.request_rw_size; From 5c1d456320908e8944af828fbdd7904d34859ddd Mon Sep 17 00:00:00 2001 From: Grzegorz Nosek Date: Fri, 16 Jul 2021 14:26:36 +0200 Subject: [PATCH 070/148] Pass container_id as a const ref to docker_lookup_request ctor Signed-off-by: Grzegorz Nosek --- userspace/libsinsp/container_engine/docker.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/userspace/libsinsp/container_engine/docker.h b/userspace/libsinsp/container_engine/docker.h index fbe472789..fad05693f 100644 --- a/userspace/libsinsp/container_engine/docker.h +++ b/userspace/libsinsp/container_engine/docker.h @@ -53,7 +53,7 @@ struct docker_lookup_request request_rw_size(false) {} - docker_lookup_request(const std::string container_id_value, + docker_lookup_request(const std::string& container_id_value, bool rw_size_value) : container_id(container_id_value), request_rw_size(rw_size_value) From 34b9818a26a6074319314b1de6492b839b5140e1 Mon Sep 17 00:00:00 2001 From: Grzegorz Nosek Date: Fri, 16 Jul 2021 14:33:08 +0200 Subject: [PATCH 071/148] Move docker_lookup_request to a separate file Signed-off-by: Grzegorz Nosek --- userspace/libsinsp/container_engine/docker.h | 33 +-------------- .../container_engine/docker/lookup_request.h | 40 +++++++++++++++++++ 2 files changed, 41 insertions(+), 32 deletions(-) create mode 100644 userspace/libsinsp/container_engine/docker/lookup_request.h diff --git a/userspace/libsinsp/container_engine/docker.h b/userspace/libsinsp/container_engine/docker.h index fad05693f..41d1b1e7a 100644 --- a/userspace/libsinsp/container_engine/docker.h +++ b/userspace/libsinsp/container_engine/docker.h @@ -37,6 +37,7 @@ limitations under the License. #include "container.h" #include "container_info.h" +#include "container_engine/docker/lookup_request.h" #include "container_engine/container_engine_base.h" #include "container_engine/sinsp_container_type.h" #include "container_engine/wmi_handle_source.h" @@ -47,38 +48,6 @@ class sinsp_threadinfo; namespace libsinsp { namespace container_engine { -struct docker_lookup_request -{ - docker_lookup_request() : - request_rw_size(false) - {} - - docker_lookup_request(const std::string& container_id_value, - bool rw_size_value) : - container_id(container_id_value), - request_rw_size(rw_size_value) - {} - - bool operator<(const docker_lookup_request& rhs) const - { - if(container_id != rhs.container_id) - { - return container_id < rhs.container_id; - } - - return request_rw_size < rhs.request_rw_size; - } - - bool operator==(const docker_lookup_request& rhs) const - { - return container_id == rhs.container_id && - request_rw_size == rhs.request_rw_size; - } - - std::string container_id; - bool request_rw_size; -}; - class docker_async_source : public sysdig::async_key_value_source { enum docker_response diff --git a/userspace/libsinsp/container_engine/docker/lookup_request.h b/userspace/libsinsp/container_engine/docker/lookup_request.h new file mode 100644 index 000000000..fa84b79f4 --- /dev/null +++ b/userspace/libsinsp/container_engine/docker/lookup_request.h @@ -0,0 +1,40 @@ +#pragma once + +namespace libsinsp { +namespace container_engine { + +struct docker_lookup_request +{ + docker_lookup_request() : + request_rw_size(false) + {} + + docker_lookup_request(const std::string& container_id_value, + bool rw_size_value) : + container_id(container_id_value), + request_rw_size(rw_size_value) + {} + + bool operator<(const docker_lookup_request& rhs) const + { + if(container_id != rhs.container_id) + { + return container_id < rhs.container_id; + } + + return request_rw_size < rhs.request_rw_size; + } + + bool operator==(const docker_lookup_request& rhs) const + { + return container_id == rhs.container_id && + request_rw_size == rhs.request_rw_size; + } + + std::string container_id; + bool request_rw_size; +}; + + +} +} From c15bf456409767d70c51fc1f077599bded27be5e Mon Sep 17 00:00:00 2001 From: Grzegorz Nosek Date: Fri, 16 Jul 2021 15:29:47 +0200 Subject: [PATCH 072/148] Pass docker socket to the async engine separately for each request Rootless podman containers have multiple sockets (one per user account), and we don't really want to spawn a separate thread for each user. This means that we can no longer use a single CURL handle but need to create a new one for each individual request. Signed-off-by: Grzegorz Nosek --- userspace/libsinsp/container_engine/docker.h | 3 +- .../container_engine/docker/lookup_request.h | 9 +++ .../container_engine/docker_common.cpp | 14 ++--- .../container_engine/docker_linux.cpp | 59 ++++++++++++------- 4 files changed, 54 insertions(+), 31 deletions(-) diff --git a/userspace/libsinsp/container_engine/docker.h b/userspace/libsinsp/container_engine/docker.h index 41d1b1e7a..8748598d6 100644 --- a/userspace/libsinsp/container_engine/docker.h +++ b/userspace/libsinsp/container_engine/docker.h @@ -75,7 +75,7 @@ class docker_async_source : public sysdig::async_key_value_sourcelookup(request, result, cb)) { // if a previous lookup call already found the metadata, process it now @@ -509,7 +508,7 @@ bool docker_async_source::parse_docker(const docker_lookup_request& request, sin api_request += "?size=true"; } - docker_response resp = get_docker(api_request, json); + docker_response resp = get_docker(request, api_request, json); switch(resp) { case docker_response::RESP_BAD_REQUEST: @@ -519,7 +518,7 @@ bool docker_async_source::parse_docker(const docker_lookup_request& request, sin m_api_version = ""; json = ""; - resp = get_docker(build_request("/containers/" + request.container_id + "/json"), json); + resp = get_docker(request, build_request("/containers/" + request.container_id + "/json"), json); if (resp == docker_response::RESP_OK) { break; @@ -604,7 +603,7 @@ bool docker_async_source::parse_docker(const docker_lookup_request& request, sin "docker_async url: %s", url.c_str()); - if(get_docker(build_request(url), img_json) == docker_response::RESP_OK) + if(get_docker(request, build_request(url), img_json) == docker_response::RESP_OK) { g_logger.format(sinsp_logger::SEV_DEBUG, "docker_async (%s) image (%s): Image info fetch returned \"%s\"", @@ -725,6 +724,7 @@ bool docker_async_source::parse_docker(const docker_lookup_request& request, sin secondary_container_id.c_str()); if(parse_docker(docker_lookup_request(secondary_container_id, + request.docker_socket, false /*don't request size since we just need the IP*/), pcnt)) { @@ -886,7 +886,7 @@ void docker::update_with_size(const std::string &container_id) container_id.c_str()); sinsp_container_info result; - docker_lookup_request request(container_id, true /*request rw size*/); + docker_lookup_request request(container_id, m_docker_sock, true /*request rw size*/); (void)m_docker_info_source->lookup(request, result, cb); } diff --git a/userspace/libsinsp/container_engine/docker_linux.cpp b/userspace/libsinsp/container_engine/docker_linux.cpp index 8122d35b9..5d3db12e3 100644 --- a/userspace/libsinsp/container_engine/docker_linux.cpp +++ b/userspace/libsinsp/container_engine/docker_linux.cpp @@ -52,33 +52,17 @@ void docker_async_source::init_docker_conn() { if(!m_curlm) { - m_curl = curl_easy_init(); m_curlm = curl_multi_init(); if(m_curlm) { curl_multi_setopt(m_curlm, CURLMOPT_PIPELINING, CURLPIPE_HTTP1|CURLPIPE_MULTIPLEX); } - - if(m_curl) - { - auto docker_path = scap_get_host_root() + m_docker_unix_socket_path; - curl_easy_setopt(m_curl, CURLOPT_UNIX_SOCKET_PATH, docker_path.c_str()); - curl_easy_setopt(m_curl, CURLOPT_HTTPGET, 1); - curl_easy_setopt(m_curl, CURLOPT_FOLLOWLOCATION, 1); - curl_easy_setopt(m_curl, CURLOPT_WRITEFUNCTION, docker_curl_write_callback); - } } } void docker_async_source::free_docker_conn() { - if(m_curl) - { - curl_easy_cleanup(m_curl); - m_curl = NULL; - } - if(m_curlm) { curl_multi_cleanup(m_curlm); @@ -91,36 +75,55 @@ std::string docker_async_source::build_request(const std::string &url) return "http://localhost" + m_api_version + url; } -docker_async_source::docker_response docker_async_source::get_docker(const std::string& url, std::string &json) +docker_async_source::docker_response docker_async_source::get_docker(const docker_lookup_request& request, const std::string& url, std::string &json) { + CURL* curl = curl_easy_init(); + if(!curl) + { + g_logger.format(sinsp_logger::SEV_WARNING, + "docker_async (%s): Failed to initialize curl handle", + request.container_id.c_str()); + return docker_response::RESP_ERROR; + } + + auto docker_path = scap_get_host_root() + request.docker_socket; + curl_easy_setopt(curl, CURLOPT_UNIX_SOCKET_PATH, docker_path.c_str()); + curl_easy_setopt(curl, CURLOPT_HTTPGET, 1); + curl_easy_setopt(curl, CURLOPT_FOLLOWLOCATION, 1); + curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, docker_curl_write_callback); g_logger.format(sinsp_logger::SEV_DEBUG, "docker_async (%s): Fetching url", url.c_str()); - if(curl_easy_setopt(m_curl, CURLOPT_URL, url.c_str()) != CURLE_OK) + if(curl_easy_setopt(curl, CURLOPT_URL, url.c_str()) != CURLE_OK) { g_logger.format(sinsp_logger::SEV_DEBUG, "docker_async (%s): curl_easy_setopt(CURLOPT_URL) failed", url.c_str()); + curl_easy_cleanup(curl); ASSERT(false); return docker_response::RESP_ERROR; } - if(curl_easy_setopt(m_curl, CURLOPT_WRITEDATA, &json) != CURLE_OK) + if(curl_easy_setopt(curl, CURLOPT_WRITEDATA, &json) != CURLE_OK) { g_logger.format(sinsp_logger::SEV_DEBUG, "docker_async (%s): curl_easy_setopt(CURLOPT_WRITEDATA) failed", url.c_str()); + curl_easy_cleanup(curl); + ASSERT(false); return docker_response::RESP_ERROR; } - if(curl_multi_add_handle(m_curlm, m_curl) != CURLM_OK) + if(curl_multi_add_handle(m_curlm, curl) != CURLM_OK) { g_logger.format(sinsp_logger::SEV_DEBUG, "docker_async (%s): curl_multi_add_handle() failed", url.c_str()); + curl_easy_cleanup(curl); + ASSERT(false); return docker_response::RESP_ERROR; } @@ -135,6 +138,9 @@ docker_async_source::docker_response docker_async_source::get_docker(const std:: "docker_async (%s): curl_multi_perform() failed", url.c_str()); + curl_multi_remove_handle(m_curlm, curl); + curl_easy_cleanup(curl); + ASSERT(false); return docker_response::RESP_ERROR; } @@ -151,31 +157,40 @@ docker_async_source::docker_response docker_async_source::get_docker(const std:: g_logger.format(sinsp_logger::SEV_DEBUG, "docker_async (%s): curl_multi_wait() failed", url.c_str()); + curl_multi_remove_handle(m_curlm, curl); + curl_easy_cleanup(curl); + ASSERT(false); return docker_response::RESP_ERROR; } } - if(curl_multi_remove_handle(m_curlm, m_curl) != CURLM_OK) + if(curl_multi_remove_handle(m_curlm, curl) != CURLM_OK) { g_logger.format(sinsp_logger::SEV_DEBUG, "docker_async (%s): curl_multi_remove_handle() failed", url.c_str()); + curl_easy_cleanup(curl); + ASSERT(false); return docker_response::RESP_ERROR; } long http_code = 0; - if(curl_easy_getinfo(m_curl, CURLINFO_RESPONSE_CODE, &http_code) != CURLE_OK) + if(curl_easy_getinfo(curl, CURLINFO_RESPONSE_CODE, &http_code) != CURLE_OK) { g_logger.format(sinsp_logger::SEV_DEBUG, "docker_async (%s): curl_easy_getinfo(CURLINFO_RESPONSE_CODE) failed", url.c_str()); + curl_easy_cleanup(curl); + ASSERT(false); return docker_response::RESP_ERROR; } + curl_easy_cleanup(curl); + g_logger.format(sinsp_logger::SEV_DEBUG, "docker_async (%s): http_code=%ld", url.c_str(), http_code); From 45941a4dbb28e778b438b2acae37d40034bc951b Mon Sep 17 00:00:00 2001 From: Grzegorz Nosek Date: Fri, 16 Jul 2021 16:12:06 +0200 Subject: [PATCH 073/148] Move platform-specific Docker-API code to a separate class All the platform-specific code from docker_async_source is moved to a new docker_connection class. The interface is almost identical, so keep a single header file with two separate implementation files (one for Linux, one for Windows). Signed-off-by: Grzegorz Nosek --- userspace/libsinsp/CMakeLists.txt | 4 +- userspace/libsinsp/container_engine/docker.h | 27 +-- .../container_engine/docker/connection.h | 46 ++++ .../docker/connection_linux.cpp | 201 ++++++++++++++++++ .../docker/connection_win.cpp | 65 ++++++ .../container_engine/docker_common.cpp | 40 +--- .../container_engine/docker_linux.cpp | 180 ---------------- .../libsinsp/container_engine/docker_win.cpp | 42 ---- 8 files changed, 328 insertions(+), 277 deletions(-) create mode 100644 userspace/libsinsp/container_engine/docker/connection.h create mode 100644 userspace/libsinsp/container_engine/docker/connection_linux.cpp create mode 100644 userspace/libsinsp/container_engine/docker/connection_win.cpp diff --git a/userspace/libsinsp/CMakeLists.txt b/userspace/libsinsp/CMakeLists.txt index 2eff18f0c..512677aab 100644 --- a/userspace/libsinsp/CMakeLists.txt +++ b/userspace/libsinsp/CMakeLists.txt @@ -142,10 +142,12 @@ if(NOT MINIMAL_BUILD) container_engine/docker_common.cpp) if(WIN32) list(APPEND SINSP_SOURCES - container_engine/docker_win.cpp) + container_engine/docker_win.cpp + container_engine/docker/connection_win.cpp) else() list(APPEND SINSP_SOURCES container_engine/docker_linux.cpp + container_engine/docker/connection_linux.cpp container_engine/libvirt_lxc.cpp container_engine/lxc.cpp container_engine/mesos.cpp diff --git a/userspace/libsinsp/container_engine/docker.h b/userspace/libsinsp/container_engine/docker.h index 8748598d6..f72866e8a 100644 --- a/userspace/libsinsp/container_engine/docker.h +++ b/userspace/libsinsp/container_engine/docker.h @@ -37,6 +37,7 @@ limitations under the License. #include "container.h" #include "container_info.h" +#include "container_engine/docker/connection.h" #include "container_engine/docker/lookup_request.h" #include "container_engine/container_engine_base.h" #include "container_engine/sinsp_container_type.h" @@ -50,19 +51,8 @@ namespace container_engine { class docker_async_source : public sysdig::async_key_value_source { - enum docker_response - { - RESP_OK = 0, - RESP_BAD_REQUEST = 1, - RESP_ERROR = 2 - }; - public: -#ifdef _WIN32 docker_async_source(uint64_t max_wait_ms, uint64_t ttl_ms, container_cache_interface *cache); -#else - docker_async_source(uint64_t max_wait_ms, uint64_t ttl_ms, container_cache_interface *cache, std::string socket_path); -#endif virtual ~docker_async_source(); static void set_query_image_info(bool query_image_info); @@ -71,12 +61,6 @@ class docker_async_source : public sysdig::async_key_value_source +#include +#include +#endif + +#include + +#include "container_engine/docker/lookup_request.h" + +namespace libsinsp { +namespace container_engine { + +class docker_connection { +public: + enum docker_response { + RESP_OK = 0, + RESP_BAD_REQUEST = 1, + RESP_ERROR = 2 + }; + + docker_connection(); + ~docker_connection(); + + std::string build_request(const std::string& url); + + docker_response + get_docker(const docker_lookup_request& request, const std::string& url, std::string& json); + + void set_api_version(const std::string& api_version) + { + m_api_version = api_version; + } + +private: + std::string m_api_version; + +#ifndef _WIN32 + CURLM *m_curlm; +#endif +}; + +} +} diff --git a/userspace/libsinsp/container_engine/docker/connection_linux.cpp b/userspace/libsinsp/container_engine/docker/connection_linux.cpp new file mode 100644 index 000000000..1f61b5fca --- /dev/null +++ b/userspace/libsinsp/container_engine/docker/connection_linux.cpp @@ -0,0 +1,201 @@ +/* +Copyright (C) 2021 The Falco Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. + +*/ +#include "connection.h" + +#include "sinsp.h" +#include "sinsp_int.h" + +namespace { + +size_t docker_curl_write_callback(const char *ptr, size_t size, size_t nmemb, std::string *json) +{ + const std::size_t total = size * nmemb; + json->append(ptr, total); + return total; +} + +} + +using namespace libsinsp::container_engine; + +docker_connection::docker_connection(): + m_api_version("/v1.24"), + m_curlm(nullptr) +{ + m_curlm = curl_multi_init(); + + if(m_curlm) + { + curl_multi_setopt(m_curlm, CURLMOPT_PIPELINING, CURLPIPE_HTTP1|CURLPIPE_MULTIPLEX); + } +} + +docker_connection::~docker_connection() +{ + if(m_curlm) + { + curl_multi_cleanup(m_curlm); + m_curlm = NULL; + } +} + +std::string docker_connection::build_request(const std::string &url) +{ + return "http://localhost" + m_api_version + url; +} + +docker_connection::docker_response docker_connection::get_docker(const docker_lookup_request& request, const std::string& url, std::string &json) +{ + CURL* curl = curl_easy_init(); + if(!curl) + { + g_logger.format(sinsp_logger::SEV_WARNING, + "docker_async (%s): Failed to initialize curl handle", + url.c_str()); + return docker_response::RESP_ERROR; + } + + auto docker_path = scap_get_host_root() + request.docker_socket; + curl_easy_setopt(curl, CURLOPT_HTTPGET, 1); + curl_easy_setopt(curl, CURLOPT_FOLLOWLOCATION, 1); + curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, docker_curl_write_callback); + curl_easy_setopt(curl, CURLOPT_UNIX_SOCKET_PATH, docker_path.c_str()); + + g_logger.format(sinsp_logger::SEV_DEBUG, + "docker_async (%s): Fetching url", + url.c_str()); + + if(curl_easy_setopt(curl, CURLOPT_URL, url.c_str()) != CURLE_OK) + { + g_logger.format(sinsp_logger::SEV_DEBUG, + "docker_async (%s): curl_easy_setopt(CURLOPT_URL) failed", + url.c_str()); + + curl_easy_cleanup(curl); + ASSERT(false); + return docker_response::RESP_ERROR; + } + if(curl_easy_setopt(curl, CURLOPT_WRITEDATA, &json) != CURLE_OK) + { + g_logger.format(sinsp_logger::SEV_DEBUG, + "docker_async (%s): curl_easy_setopt(CURLOPT_WRITEDATA) failed", + url.c_str()); + curl_easy_cleanup(curl); + ASSERT(false); + return docker_response::RESP_ERROR; + } + + if(curl_multi_add_handle(m_curlm, curl) != CURLM_OK) + { + g_logger.format(sinsp_logger::SEV_DEBUG, + "docker_async (%s): curl_multi_add_handle() failed", + url.c_str()); + curl_easy_cleanup(curl); + ASSERT(false); + return docker_response::RESP_ERROR; + } + + while(true) + { + int still_running; + CURLMcode res = curl_multi_perform(m_curlm, &still_running); + if(res != CURLM_OK) + { + g_logger.format(sinsp_logger::SEV_DEBUG, + "docker_async (%s): curl_multi_perform() failed", + url.c_str()); + + curl_multi_remove_handle(m_curlm, curl); + curl_easy_cleanup(curl); + ASSERT(false); + return docker_response::RESP_ERROR; + } + + if(still_running == 0) + { + break; + } + + int numfds; + res = curl_multi_wait(m_curlm, NULL, 0, 1000, &numfds); + if(res != CURLM_OK) + { + g_logger.format(sinsp_logger::SEV_DEBUG, + "docker_async (%s): curl_multi_wait() failed", + url.c_str()); + + curl_multi_remove_handle(m_curlm, curl); + curl_easy_cleanup(curl); + ASSERT(false); + return docker_response::RESP_ERROR; + } + } + + if(curl_multi_remove_handle(m_curlm, curl) != CURLM_OK) + { + g_logger.format(sinsp_logger::SEV_DEBUG, + "docker_async (%s): curl_multi_remove_handle() failed", + url.c_str()); + + curl_easy_cleanup(curl); + ASSERT(false); + return docker_response::RESP_ERROR; + } + + long http_code = 0; + if(curl_easy_getinfo(curl, CURLINFO_RESPONSE_CODE, &http_code) != CURLE_OK) + { + g_logger.format(sinsp_logger::SEV_DEBUG, + "docker_async (%s): curl_easy_getinfo(CURLINFO_RESPONSE_CODE) failed", + url.c_str()); + + curl_easy_cleanup(curl); + ASSERT(false); + return docker_response::RESP_ERROR; + } + + curl_easy_cleanup(curl); + g_logger.format(sinsp_logger::SEV_DEBUG, + "docker_async (%s): http_code=%ld", + url.c_str(), http_code); + + switch(http_code) + { + case 0: /* connection failed, apparently */ + g_logger.format(sinsp_logger::SEV_DEBUG, + "docker_async (%s): returning RESP_ERROR", + url.c_str()); + return docker_response::RESP_ERROR; + case 200: + g_logger.format(sinsp_logger::SEV_DEBUG, + "docker_async (%s): returning RESP_OK", + url.c_str()); + return docker_response::RESP_OK; + default: + g_logger.format(sinsp_logger::SEV_DEBUG, + "docker_async (%s): returning RESP_BAD_REQUEST", + url.c_str()); + return docker_response::RESP_BAD_REQUEST; + } + + g_logger.format(sinsp_logger::SEV_DEBUG, + "docker_async (%s): fallthrough, returning RESP_OK", + url.c_str()); + + return docker_response::RESP_OK; +} + diff --git a/userspace/libsinsp/container_engine/docker/connection_win.cpp b/userspace/libsinsp/container_engine/docker/connection_win.cpp new file mode 100644 index 000000000..218f863b5 --- /dev/null +++ b/userspace/libsinsp/container_engine/docker/connection_win.cpp @@ -0,0 +1,65 @@ +/* +Copyright (C) 2021 The Falco Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. + +*/ +#include "connection.h" + +#include "dragent_win_hal_public.h" + +using namespace libsinsp::container_engine; + +docker_connection::docker_connection(): + m_api_version("/v1.30") +{ +} + +docker_connection::~docker_connection() +{ +} + +std::string docker_connection::build_request(const std::string &url) +{ + return "GET " + m_api_version + url + " HTTP/1.1\r\nHost: docker\r\n\r\n"; +} + +docker_connection::docker_response docker_connection::get_docker(const docker_lookup_request& request, const std::string& url, std::string &json) +{ + const char* response = NULL; + bool qdres = wh_query_docker(m_inspector->get_wmi_handle(), + (char*)url.c_str(), + &response); + if(qdres == false) + { + ASSERT(false); + return docker_response::RESP_ERROR; + } + + json = response; + if(strncmp(json.c_str(), "HTTP/1.0 200 OK", sizeof("HTTP/1.0 200 OK") -1)) + { + return docker_response::RESP_BAD_REQUEST; + } + + size_t pos = json.find("{"); + if(pos == string::npos) + { + ASSERT(false); + return docker_response::RESP_ERROR; + } + json = json.substr(pos); + + return docker_response::RESP_OK; +} + diff --git a/userspace/libsinsp/container_engine/docker_common.cpp b/userspace/libsinsp/container_engine/docker_common.cpp index 88c527a0b..078e39ab6 100644 --- a/userspace/libsinsp/container_engine/docker_common.cpp +++ b/userspace/libsinsp/container_engine/docker_common.cpp @@ -29,22 +29,10 @@ using namespace libsinsp::container_engine; docker_async_source::docker_async_source(uint64_t max_wait_ms, uint64_t ttl_ms, - container_cache_interface *cache -#ifndef _WIN32 - , std::string socket_path -#endif - ) + container_cache_interface *cache) : async_key_value_source(max_wait_ms, ttl_ms), - m_cache(cache), -#ifdef _WIN32 - m_api_version("/v1.30") -#else - m_api_version("/v1.24"), - m_docker_unix_socket_path(std::move(socket_path)), - m_curlm(NULL) -#endif + m_cache(cache) { - init_docker_conn(); } docker_async_source::~docker_async_source() @@ -52,8 +40,6 @@ docker_async_source::~docker_async_source() this->stop(); g_logger.format(sinsp_logger::SEV_DEBUG, "docker_async: Source destructor"); - - free_docker_conn(); } void docker_async_source::run_impl() @@ -423,11 +409,7 @@ bool docker::resolve(sinsp_threadinfo *tinfo, bool query_os_for_missing_info) g_logger.log("docker_async: Creating docker async source", sinsp_logger::SEV_DEBUG); uint64_t max_wait_ms = 10000; -#ifdef _WIN32 docker_async_source *src = new docker_async_source(docker_async_source::NO_WAIT_LOOKUP, max_wait_ms, cache); -#else - docker_async_source *src = new docker_async_source(docker_async_source::NO_WAIT_LOOKUP, max_wait_ms, cache, m_docker_sock); -#endif m_docker_info_source.reset(src); } @@ -502,36 +484,36 @@ bool docker_async_source::parse_docker(const docker_lookup_request& request, sin "docker_async (%s): Looking up info for container", request.container_id.c_str()); - std::string api_request = build_request("/containers/" + request.container_id + "/json"); + std::string api_request = m_connection.build_request("/containers/" + request.container_id + "/json"); if(request.request_rw_size) { api_request += "?size=true"; } - docker_response resp = get_docker(request, api_request, json); + docker_connection::docker_response resp = m_connection.get_docker(request, api_request, json); switch(resp) { - case docker_response::RESP_BAD_REQUEST: + case docker_connection::docker_response::RESP_BAD_REQUEST: g_logger.format(sinsp_logger::SEV_DEBUG, "docker_async (%s): Initial url fetch failed, trying w/o api version", request.container_id.c_str()); - m_api_version = ""; + m_connection.set_api_version(""); json = ""; - resp = get_docker(request, build_request("/containers/" + request.container_id + "/json"), json); - if (resp == docker_response::RESP_OK) + resp = m_connection.get_docker(request, m_connection.build_request("/containers/" + request.container_id + "/json"), json); + if (resp == docker_connection::docker_response::RESP_OK) { break; } /* FALLTHRU */ - case docker_response::RESP_ERROR: + case docker_connection::docker_response::RESP_ERROR: g_logger.format(sinsp_logger::SEV_DEBUG, "docker_async (%s): Url fetch failed, returning false", request.container_id.c_str()); return false; - case docker_response::RESP_OK: + case docker_connection::docker_response::RESP_OK: break; } @@ -603,7 +585,7 @@ bool docker_async_source::parse_docker(const docker_lookup_request& request, sin "docker_async url: %s", url.c_str()); - if(get_docker(request, build_request(url), img_json) == docker_response::RESP_OK) + if(m_connection.get_docker(request, m_connection.build_request(url), img_json) == docker_connection::docker_response::RESP_OK) { g_logger.format(sinsp_logger::SEV_DEBUG, "docker_async (%s) image (%s): Image info fetch returned \"%s\"", diff --git a/userspace/libsinsp/container_engine/docker_linux.cpp b/userspace/libsinsp/container_engine/docker_linux.cpp index 5d3db12e3..2648de922 100644 --- a/userspace/libsinsp/container_engine/docker_linux.cpp +++ b/userspace/libsinsp/container_engine/docker_linux.cpp @@ -27,13 +27,6 @@ using namespace libsinsp::runc; namespace { -size_t docker_curl_write_callback(const char* ptr, size_t size, size_t nmemb, string* json) -{ - const std::size_t total = size * nmemb; - json->append(ptr, total); - return total; -} - constexpr const cgroup_layout DOCKER_CGROUP_LAYOUT[] = { {"/", ""}, // non-systemd docker {"/docker-", ".scope"}, // systemd docker @@ -48,179 +41,6 @@ void docker::cleanup() m_docker_info_source.reset(NULL); } -void docker_async_source::init_docker_conn() -{ - if(!m_curlm) - { - m_curlm = curl_multi_init(); - - if(m_curlm) - { - curl_multi_setopt(m_curlm, CURLMOPT_PIPELINING, CURLPIPE_HTTP1|CURLPIPE_MULTIPLEX); - } - } -} - -void docker_async_source::free_docker_conn() -{ - if(m_curlm) - { - curl_multi_cleanup(m_curlm); - m_curlm = NULL; - } -} - -std::string docker_async_source::build_request(const std::string &url) -{ - return "http://localhost" + m_api_version + url; -} - -docker_async_source::docker_response docker_async_source::get_docker(const docker_lookup_request& request, const std::string& url, std::string &json) -{ - CURL* curl = curl_easy_init(); - if(!curl) - { - g_logger.format(sinsp_logger::SEV_WARNING, - "docker_async (%s): Failed to initialize curl handle", - request.container_id.c_str()); - return docker_response::RESP_ERROR; - } - - auto docker_path = scap_get_host_root() + request.docker_socket; - curl_easy_setopt(curl, CURLOPT_UNIX_SOCKET_PATH, docker_path.c_str()); - curl_easy_setopt(curl, CURLOPT_HTTPGET, 1); - curl_easy_setopt(curl, CURLOPT_FOLLOWLOCATION, 1); - curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, docker_curl_write_callback); - - g_logger.format(sinsp_logger::SEV_DEBUG, - "docker_async (%s): Fetching url", - url.c_str()); - - if(curl_easy_setopt(curl, CURLOPT_URL, url.c_str()) != CURLE_OK) - { - g_logger.format(sinsp_logger::SEV_DEBUG, - "docker_async (%s): curl_easy_setopt(CURLOPT_URL) failed", - url.c_str()); - curl_easy_cleanup(curl); - - ASSERT(false); - return docker_response::RESP_ERROR; - } - if(curl_easy_setopt(curl, CURLOPT_WRITEDATA, &json) != CURLE_OK) - { - g_logger.format(sinsp_logger::SEV_DEBUG, - "docker_async (%s): curl_easy_setopt(CURLOPT_WRITEDATA) failed", - url.c_str()); - curl_easy_cleanup(curl); - - ASSERT(false); - return docker_response::RESP_ERROR; - } - - if(curl_multi_add_handle(m_curlm, curl) != CURLM_OK) - { - g_logger.format(sinsp_logger::SEV_DEBUG, - "docker_async (%s): curl_multi_add_handle() failed", - url.c_str()); - curl_easy_cleanup(curl); - - ASSERT(false); - return docker_response::RESP_ERROR; - } - - while(true) - { - int still_running; - CURLMcode res = curl_multi_perform(m_curlm, &still_running); - if(res != CURLM_OK) - { - g_logger.format(sinsp_logger::SEV_DEBUG, - "docker_async (%s): curl_multi_perform() failed", - url.c_str()); - - curl_multi_remove_handle(m_curlm, curl); - curl_easy_cleanup(curl); - - ASSERT(false); - return docker_response::RESP_ERROR; - } - - if(still_running == 0) - { - break; - } - - int numfds; - res = curl_multi_wait(m_curlm, NULL, 0, 1000, &numfds); - if(res != CURLM_OK) - { - g_logger.format(sinsp_logger::SEV_DEBUG, - "docker_async (%s): curl_multi_wait() failed", - url.c_str()); - curl_multi_remove_handle(m_curlm, curl); - curl_easy_cleanup(curl); - - ASSERT(false); - return docker_response::RESP_ERROR; - } - } - - if(curl_multi_remove_handle(m_curlm, curl) != CURLM_OK) - { - g_logger.format(sinsp_logger::SEV_DEBUG, - "docker_async (%s): curl_multi_remove_handle() failed", - url.c_str()); - - curl_easy_cleanup(curl); - - ASSERT(false); - return docker_response::RESP_ERROR; - } - - long http_code = 0; - if(curl_easy_getinfo(curl, CURLINFO_RESPONSE_CODE, &http_code) != CURLE_OK) - { - g_logger.format(sinsp_logger::SEV_DEBUG, - "docker_async (%s): curl_easy_getinfo(CURLINFO_RESPONSE_CODE) failed", - url.c_str()); - curl_easy_cleanup(curl); - - ASSERT(false); - return docker_response::RESP_ERROR; - } - - curl_easy_cleanup(curl); - - g_logger.format(sinsp_logger::SEV_DEBUG, - "docker_async (%s): http_code=%ld", - url.c_str(), http_code); - - switch(http_code) - { - case 0: /* connection failed, apparently */ - g_logger.format(sinsp_logger::SEV_DEBUG, - "docker_async (%s): returning RESP_ERROR", - url.c_str()); - return docker_response::RESP_ERROR; - case 200: - g_logger.format(sinsp_logger::SEV_DEBUG, - "docker_async (%s): returning RESP_OK", - url.c_str()); - return docker_response::RESP_OK; - default: - g_logger.format(sinsp_logger::SEV_DEBUG, - "docker_async (%s): returning RESP_BAD_REQUEST", - url.c_str()); - return docker_response::RESP_BAD_REQUEST; - } - - g_logger.format(sinsp_logger::SEV_DEBUG, - "docker_async (%s): fallthrough, returning RESP_OK", - url.c_str()); - - return docker_response::RESP_OK; -} - bool docker::detect_docker(const sinsp_threadinfo *tinfo, std::string &container_id, std::string &container_name) { if(matches_runc_cgroups(tinfo, DOCKER_CGROUP_LAYOUT, container_id)) diff --git a/userspace/libsinsp/container_engine/docker_win.cpp b/userspace/libsinsp/container_engine/docker_win.cpp index 6be4c4521..965e792d7 100644 --- a/userspace/libsinsp/container_engine/docker_win.cpp +++ b/userspace/libsinsp/container_engine/docker_win.cpp @@ -34,19 +34,6 @@ void docker::cleanup() g_docker_info_source.reset(NULL); } -void docker_async_source::init_docker_conn() -{ -} - -void docker_async_source::free_docker_conn() -{ -} - -std::string docker_async_source::build_request(const std::string &url) -{ - return "GET " + m_api_version + url + " HTTP/1.1\r\nHost: docker\r\n\r\n"; -} - bool docker::detect_docker(sinsp_threadinfo *tinfo, std::string &container_id, std::string &container_name) { wh_docker_container_info wcinfo = wh_docker_resolve_pid(m_wmi_handle_source.get_wmi_handle(), tinfo->m_pid); @@ -61,33 +48,4 @@ bool docker::detect_docker(sinsp_threadinfo *tinfo, std::string &container_id, s return true; } -docker_async_source::docker_response docker_async_source::get_docker(const std::string& url, std::string &json) -{ - const char* response = NULL; - bool qdres = wh_query_docker(m_inspector->get_wmi_handle(), - (char*)url.c_str(), - &response); - if(qdres == false) - { - ASSERT(false); - return docker_response::RESP_ERROR; - } - - json = response; - if(strncmp(json.c_str(), "HTTP/1.0 200 OK", sizeof("HTTP/1.0 200 OK") -1)) - { - return docker_response::RESP_BAD_REQUEST; - } - - size_t pos = json.find("{"); - if(pos == string::npos) - { - ASSERT(false); - return docker_response::RESP_ERROR; - } - json = json.substr(pos); - - return docker_response::RESP_OK; -} - #endif // CYGWING_AGENT From 3ef184947e3c54dcd86cbd274798945c4ac566ab Mon Sep 17 00:00:00 2001 From: Grzegorz Nosek Date: Fri, 16 Jul 2021 16:22:56 +0200 Subject: [PATCH 074/148] Remove docker_connection::build_request This is a private implementation detail of the connection class. Also, this fixes a but where (on Windows) we appended `?size=true` to the wrong place in the request (to the Host header, not the URL). Signed-off-by: Grzegorz Nosek --- .../libsinsp/container_engine/docker/connection.h | 4 +--- .../container_engine/docker/connection_linux.cpp | 11 ++++------- .../container_engine/docker/connection_win.cpp | 9 +++------ userspace/libsinsp/container_engine/docker_common.cpp | 6 +++--- 4 files changed, 11 insertions(+), 19 deletions(-) diff --git a/userspace/libsinsp/container_engine/docker/connection.h b/userspace/libsinsp/container_engine/docker/connection.h index 0ffd40ee1..796631abc 100644 --- a/userspace/libsinsp/container_engine/docker/connection.h +++ b/userspace/libsinsp/container_engine/docker/connection.h @@ -24,10 +24,8 @@ class docker_connection { docker_connection(); ~docker_connection(); - std::string build_request(const std::string& url); - docker_response - get_docker(const docker_lookup_request& request, const std::string& url, std::string& json); + get_docker(const docker_lookup_request& request, const std::string& req_url, std::string& json); void set_api_version(const std::string& api_version) { diff --git a/userspace/libsinsp/container_engine/docker/connection_linux.cpp b/userspace/libsinsp/container_engine/docker/connection_linux.cpp index 1f61b5fca..fba584d14 100644 --- a/userspace/libsinsp/container_engine/docker/connection_linux.cpp +++ b/userspace/libsinsp/container_engine/docker/connection_linux.cpp @@ -53,19 +53,14 @@ docker_connection::~docker_connection() } } -std::string docker_connection::build_request(const std::string &url) -{ - return "http://localhost" + m_api_version + url; -} - -docker_connection::docker_response docker_connection::get_docker(const docker_lookup_request& request, const std::string& url, std::string &json) +docker_connection::docker_response docker_connection::get_docker(const docker_lookup_request& request, const std::string& req_url, std::string &json) { CURL* curl = curl_easy_init(); if(!curl) { g_logger.format(sinsp_logger::SEV_WARNING, "docker_async (%s): Failed to initialize curl handle", - url.c_str()); + req_url.c_str()); return docker_response::RESP_ERROR; } @@ -75,6 +70,8 @@ docker_connection::docker_response docker_connection::get_docker(const docker_lo curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, docker_curl_write_callback); curl_easy_setopt(curl, CURLOPT_UNIX_SOCKET_PATH, docker_path.c_str()); + std::string url = "http://localhost" + m_api_version + req_url; + g_logger.format(sinsp_logger::SEV_DEBUG, "docker_async (%s): Fetching url", url.c_str()); diff --git a/userspace/libsinsp/container_engine/docker/connection_win.cpp b/userspace/libsinsp/container_engine/docker/connection_win.cpp index 218f863b5..ee00954ed 100644 --- a/userspace/libsinsp/container_engine/docker/connection_win.cpp +++ b/userspace/libsinsp/container_engine/docker/connection_win.cpp @@ -29,16 +29,13 @@ docker_connection::~docker_connection() { } -std::string docker_connection::build_request(const std::string &url) +docker_connection::docker_response docker_connection::get_docker(const docker_lookup_request& request, const std::string& req_url, std::string &json) { - return "GET " + m_api_version + url + " HTTP/1.1\r\nHost: docker\r\n\r\n"; -} + std::string req = "GET " + m_api_version + req_url + " HTTP/1.1\r\nHost: docker\r\n\r\n"; -docker_connection::docker_response docker_connection::get_docker(const docker_lookup_request& request, const std::string& url, std::string &json) -{ const char* response = NULL; bool qdres = wh_query_docker(m_inspector->get_wmi_handle(), - (char*)url.c_str(), + (char*)req.c_str(), &response); if(qdres == false) { diff --git a/userspace/libsinsp/container_engine/docker_common.cpp b/userspace/libsinsp/container_engine/docker_common.cpp index 078e39ab6..fd9e788bc 100644 --- a/userspace/libsinsp/container_engine/docker_common.cpp +++ b/userspace/libsinsp/container_engine/docker_common.cpp @@ -484,7 +484,7 @@ bool docker_async_source::parse_docker(const docker_lookup_request& request, sin "docker_async (%s): Looking up info for container", request.container_id.c_str()); - std::string api_request = m_connection.build_request("/containers/" + request.container_id + "/json"); + std::string api_request = "/containers/" + request.container_id + "/json"; if(request.request_rw_size) { api_request += "?size=true"; @@ -500,7 +500,7 @@ bool docker_async_source::parse_docker(const docker_lookup_request& request, sin m_connection.set_api_version(""); json = ""; - resp = m_connection.get_docker(request, m_connection.build_request("/containers/" + request.container_id + "/json"), json); + resp = m_connection.get_docker(request, "/containers/" + request.container_id + "/json", json); if (resp == docker_connection::docker_response::RESP_OK) { break; @@ -585,7 +585,7 @@ bool docker_async_source::parse_docker(const docker_lookup_request& request, sin "docker_async url: %s", url.c_str()); - if(m_connection.get_docker(request, m_connection.build_request(url), img_json) == docker_connection::docker_response::RESP_OK) + if(m_connection.get_docker(request, url, img_json) == docker_connection::docker_response::RESP_OK) { g_logger.format(sinsp_logger::SEV_DEBUG, "docker_async (%s) image (%s): Image info fetch returned \"%s\"", From 48338797598c7d0f7ccad4aeaa93ba739829f248 Mon Sep 17 00:00:00 2001 From: Grzegorz Nosek Date: Fri, 16 Jul 2021 16:56:43 +0200 Subject: [PATCH 075/148] Move docker_async_source to its own source file Signed-off-by: Grzegorz Nosek --- userspace/libsinsp/CMakeLists.txt | 3 +- userspace/libsinsp/container_engine/docker.h | 57 +- .../container_engine/docker/async_source.cpp | 768 ++++++++++++++++++ .../container_engine/docker/async_source.h | 72 ++ .../container_engine/docker_common.cpp | 739 ----------------- userspace/libsinsp/parsers.cpp | 2 +- 6 files changed, 844 insertions(+), 797 deletions(-) create mode 100644 userspace/libsinsp/container_engine/docker/async_source.cpp create mode 100644 userspace/libsinsp/container_engine/docker/async_source.h diff --git a/userspace/libsinsp/CMakeLists.txt b/userspace/libsinsp/CMakeLists.txt index 512677aab..87be1bf56 100644 --- a/userspace/libsinsp/CMakeLists.txt +++ b/userspace/libsinsp/CMakeLists.txt @@ -139,7 +139,8 @@ if(NOT MINIMAL_BUILD) mesos_http.cpp mesos_state.cpp sinsp_curl.cpp - container_engine/docker_common.cpp) + container_engine/docker_common.cpp + container_engine/docker/async_source.cpp) if(WIN32) list(APPEND SINSP_SOURCES container_engine/docker_win.cpp diff --git a/userspace/libsinsp/container_engine/docker.h b/userspace/libsinsp/container_engine/docker.h index f72866e8a..99a4662d0 100644 --- a/userspace/libsinsp/container_engine/docker.h +++ b/userspace/libsinsp/container_engine/docker.h @@ -37,6 +37,7 @@ limitations under the License. #include "container.h" #include "container_info.h" +#include "container_engine/docker/async_source.h" #include "container_engine/docker/connection.h" #include "container_engine/docker/lookup_request.h" #include "container_engine/container_engine_base.h" @@ -49,61 +50,6 @@ class sinsp_threadinfo; namespace libsinsp { namespace container_engine { -class docker_async_source : public sysdig::async_key_value_source -{ -public: - docker_async_source(uint64_t max_wait_ms, uint64_t ttl_ms, container_cache_interface *cache); - virtual ~docker_async_source(); - - static void set_query_image_info(bool query_image_info); - -protected: - void run_impl(); - -private: - bool parse_docker(const docker_lookup_request& request, sinsp_container_info& container); - - // Look for a pod specification in this container's labels and - // if found set spec to the pod spec. - bool get_k8s_pod_spec(const Json::Value &config_obj, - Json::Value &spec); - - std::string normalize_arg(const std::string &arg); - - // Parse a healthcheck out of the provided healthcheck object, - // updating the container info with any healthcheck found. - void parse_healthcheck(const Json::Value &healthcheck_obj, - sinsp_container_info &container); - - // Parse either a readiness or liveness probe out of the - // provided object, updating the container info with any probe - // found. Returns true if the healthcheck/livenesss/readiness - // probe info was found and could be parsed. - bool parse_liveness_readiness_probe(const Json::Value &probe_obj, - sinsp_container_info::container_health_probe::probe_type ptype, - sinsp_container_info &container); - - // See if this config has a io.kubernetes.sandbox.id label - // referring to a different container. (NOTE: this is not the - // same as docker's sandbox id, which refers to networks.) If - // it does, try to copy the health checks from that container - // to the provided container_info pointer. Returns true if a - // sandbox container id was found, the corresponding container - // was found, and if the health checks could be copied from - // that container. - bool get_sandbox_liveness_readiness_probes(const Json::Value &config_obj, - sinsp_container_info &container); - - // Parse all healthchecks/liveness probes/readiness probes out - // of the provided object, updating the container info as required. - void parse_health_probes(const Json::Value &config_obj, - sinsp_container_info &container); - - container_cache_interface *m_cache; - docker_connection m_connection; - static bool m_query_image_info; -}; - class docker : public container_engine_base { public: @@ -115,7 +61,6 @@ class docker : public container_engine_base {} #endif void cleanup() override; - static void parse_json_mounts(const Json::Value &mnt_obj, std::vector &mounts); // Container name only set for windows. For linux name must be fetched via lookup static bool detect_docker(const sinsp_threadinfo* tinfo, std::string& container_id, std::string &container_name); diff --git a/userspace/libsinsp/container_engine/docker/async_source.cpp b/userspace/libsinsp/container_engine/docker/async_source.cpp new file mode 100644 index 000000000..fa36c7828 --- /dev/null +++ b/userspace/libsinsp/container_engine/docker/async_source.cpp @@ -0,0 +1,768 @@ +/* +Copyright (C) 2021 The Falco Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. + +*/ +#include "async_source.h" +#include "cgroup_list_counter.h" +#include "sinsp.h" +#include "sinsp_int.h" +#include "container.h" +#include "utils.h" +#include + +using namespace libsinsp::container_engine; + +bool docker_async_source::m_query_image_info = true; + +docker_async_source::docker_async_source(uint64_t max_wait_ms, + uint64_t ttl_ms, + container_cache_interface *cache) + : async_key_value_source(max_wait_ms, ttl_ms), + m_cache(cache) +{ +} + +docker_async_source::~docker_async_source() +{ + this->stop(); + g_logger.format(sinsp_logger::SEV_DEBUG, + "docker_async: Source destructor"); +} + +void docker_async_source::run_impl() +{ + docker_lookup_request request; + + while (dequeue_next_key(request)) + { + g_logger.format(sinsp_logger::SEV_DEBUG, + "docker_async (%s : %s): Source dequeued key", + request.container_id.c_str(), + request.request_rw_size ? "true" : "false"); + + sinsp_container_info res; + + res.m_lookup_state = sinsp_container_lookup_state::SUCCESSFUL; + res.m_type = CT_DOCKER; + res.m_id = request.container_id; + + if(!parse_docker(request, res)) + { + // This is not always an error e.g. when using + // containerd as the runtime. Since the cgroup + // names are often identical between + // containerd and docker, we have to try to + // fetch both. + g_logger.format(sinsp_logger::SEV_DEBUG, + "docker_async (%s): Failed to get Docker metadata, returning successful=false", + request.container_id.c_str()); + res.m_lookup_state = sinsp_container_lookup_state::FAILED; + } + + g_logger.format(sinsp_logger::SEV_DEBUG, + "docker_async (%s): Parse successful, storing value", + request.container_id.c_str()); + + // Return a result object either way, to ensure any + // new container callbacks are called. + store_value(request, res); + } +} + +bool docker_async_source::get_k8s_pod_spec(const Json::Value &config_obj, + Json::Value &spec) +{ + std::string cfg_str; + Json::Reader reader; + std::string k8s_label = "annotation.kubectl.kubernetes.io/last-applied-configuration"; + + if(config_obj.isNull() || + !config_obj.isMember("Labels") || + !config_obj["Labels"].isMember(k8s_label)) + { + return false; + } + + // The pod spec is stored as a stringified json label on the container + cfg_str = config_obj["Labels"][k8s_label].asString(); + + if(cfg_str == "") + { + return false; + } + + Json::Value cfg; + if(!reader.parse(cfg_str.c_str(), cfg)) + { + g_logger.format(sinsp_logger::SEV_WARNING, "Could not parse pod config '%s'", cfg_str.c_str()); + return false; + } + + if(!cfg.isMember("spec") || + !cfg["spec"].isMember("containers") || + !cfg["spec"]["containers"].isArray()) + { + return false; + } + + // XXX/mstemm how will this work with init containers? + spec = cfg["spec"]["containers"][0]; + + return true; +} + +std::string docker_async_source::normalize_arg(const std::string &arg) +{ + std::string ret = arg; + + if(ret.empty()) + { + return ret; + } + + // Remove pairs of leading/trailing " or ' chars, if present + while(ret.front() == '"' || ret.front() == '\'') + { + if(ret.back() == ret.front()) + { + ret.pop_back(); + ret.erase(0, 1); + } + } + + return ret; +} + +void docker_async_source::parse_healthcheck(const Json::Value &healthcheck_obj, + sinsp_container_info &container) +{ + g_logger.format(sinsp_logger::SEV_DEBUG, + "docker (%s): Trying to parse healthcheck from %s", + container.m_id.c_str(), Json::FastWriter().write(healthcheck_obj).c_str()); + + if(healthcheck_obj.isNull()) + { + g_logger.format(sinsp_logger::SEV_WARNING, "Could not parse health check from %s (No Healthcheck property)", + Json::FastWriter().write(healthcheck_obj).c_str()); + + return; + } + + if(!healthcheck_obj.isMember("Test")) + { + g_logger.format(sinsp_logger::SEV_WARNING, "Could not parse health check from %s (Healthcheck does not have Test property)", + Json::FastWriter().write(healthcheck_obj).c_str()); + + return; + } + + const Json::Value &test_obj = healthcheck_obj["Test"]; + + if(!test_obj.isArray()) + { + g_logger.format(sinsp_logger::SEV_WARNING, "Could not parse health check from %s (Healthcheck Test property is not array)", + Json::FastWriter().write(healthcheck_obj).c_str()); + return; + } + + if(test_obj.size() == 1) + { + if(test_obj[0].asString() != "NONE") + { + g_logger.format(sinsp_logger::SEV_WARNING, "Could not parse health check from %s (Expected NONE for single-element Test array)", + Json::FastWriter().write(healthcheck_obj).c_str()); + } + return; + } + + if(test_obj[0].asString() == "CMD") + { + std::string exe = normalize_arg(test_obj[1].asString()); + std::vector args; + + for(uint32_t i = 2; i < test_obj.size(); i++) + { + args.push_back(normalize_arg(test_obj[i].asString())); + } + + g_logger.format(sinsp_logger::SEV_DEBUG, + "docker (%s): Setting PT_HEALTHCHECK exe=%s nargs=%d", + container.m_id.c_str(), exe.c_str(), args.size()); + + container.m_health_probes.emplace_back(sinsp_container_info::container_health_probe::PT_HEALTHCHECK, + std::move(exe), + std::move(args)); + } + else if(test_obj[0].asString() == "CMD-SHELL") + { + std::string exe = "/bin/sh"; + std::vector args; + + args.push_back("-c"); + args.push_back(test_obj[1].asString()); + + g_logger.format(sinsp_logger::SEV_DEBUG, + "docker (%s): Setting PT_HEALTHCHECK exe=%s nargs=%d", + container.m_id.c_str(), exe.c_str(), args.size()); + + container.m_health_probes.emplace_back(sinsp_container_info::container_health_probe::PT_HEALTHCHECK, + std::move(exe), + std::move(args)); + } + else + { + g_logger.format(sinsp_logger::SEV_WARNING, "Could not parse health check from %s (Expected CMD/CMD-SHELL for multi-element Test array)", + Json::FastWriter().write(healthcheck_obj).c_str()); + return; + } +} + +bool docker_async_source::parse_liveness_readiness_probe(const Json::Value &probe_obj, + sinsp_container_info::container_health_probe::probe_type ptype, + sinsp_container_info &container) +{ + if(probe_obj.isNull() || + !probe_obj.isMember("exec") || + !probe_obj["exec"].isMember("command")) + { + g_logger.format(sinsp_logger::SEV_WARNING, "Could not parse liveness/readiness probe from %s", + Json::FastWriter().write(probe_obj).c_str()); + return false; + } + + const Json::Value command_obj = probe_obj["exec"]["command"]; + + if(!command_obj.isNull() && command_obj.isArray()) + { + std::string exe; + std::vector args; + + exe = normalize_arg(command_obj[0].asString()); + for(uint32_t i = 1; i < command_obj.size(); i++) + { + args.push_back(normalize_arg(command_obj[i].asString())); + } + + g_logger.format(sinsp_logger::SEV_DEBUG, + "docker (%s): Setting %s exe=%s nargs=%d", + container.m_id.c_str(), + sinsp_container_info::container_health_probe::probe_type_names[ptype].c_str(), + exe.c_str(), args.size()); + + container.m_health_probes.emplace_back(ptype, std::move(exe), std::move(args)); + } + + return true; +} + +bool docker_async_source::get_sandbox_liveness_readiness_probes(const Json::Value &config_obj, + sinsp_container_info &container) +{ + std::string sandbox_container_id; + std::string sandbox_label = "io.kubernetes.sandbox.id"; + + if(config_obj.isNull() || + !config_obj.isMember("Labels") || + !config_obj["Labels"].isMember(sandbox_label)) + { + SINSP_DEBUG("docker (%s): No sandbox label found, not copying liveness/readiness probes", + container.m_id.c_str()); + return false; + } + + sandbox_container_id = config_obj["Labels"][sandbox_label].asString(); + + if(sandbox_container_id.size() > 12) + { + sandbox_container_id.resize(12); + } + + sinsp_container_info::ptr_t sandbox_container = m_cache->get_container(sandbox_container_id); + + if(!sandbox_container) + { + SINSP_DEBUG("docker (%s): Sandbox container %s doesn't exist, not copying liveness/readiness probes", + container.m_id.c_str(), sandbox_container_id.c_str()); + return false; + } + + if(sandbox_container->m_health_probes.size() == 0) + { + SINSP_DEBUG("docker (%s): Sandbox container %s has no liveness/readiness probes, not copying", + container.m_id.c_str(), sandbox_container_id.c_str()); + return false; + } + + SINSP_DEBUG("docker (%s): Copying liveness/readiness probes from sandbox container %s", + container.m_id.c_str(), sandbox_container_id.c_str()); + container.m_health_probes = sandbox_container->m_health_probes; + + return true; +} + +void docker_async_source::parse_health_probes(const Json::Value &config_obj, + sinsp_container_info &container) +{ + Json::Value spec; + bool liveness_readiness_added = false; + + // When parsing the full container json for live containers, a label contains stringified json that + // contains the probes. + if (get_k8s_pod_spec(config_obj, spec)) + { + g_logger.format(sinsp_logger::SEV_DEBUG, + "docker (%s): Parsing liveness/readiness probes from pod spec", + container.m_id.c_str()); + + if(spec.isMember("livenessProbe")) + { + if(parse_liveness_readiness_probe(spec["livenessProbe"], + sinsp_container_info::container_health_probe::PT_LIVENESS_PROBE, + container)) + { + liveness_readiness_added = true; + } + } + else if(spec.isMember("readinessProbe")) + { + if(parse_liveness_readiness_probe(spec["readinessProbe"], + sinsp_container_info::container_health_probe::PT_READINESS_PROBE, + container)) + { + liveness_readiness_added = true; + } + } + } + // Otherwise, try to copy the liveness/readiness probe from the sandbox container, if it exists. + else if (get_sandbox_liveness_readiness_probes(config_obj, container)) + { + liveness_readiness_added = true; + } + else + { + g_logger.format(sinsp_logger::SEV_DEBUG, + "docker (%s): No liveness/readiness probes found", + container.m_id.c_str()); + } + + // To avoid any confusion about containers that both refer to + // a healthcheck and liveness/readiness probe, we only + // consider a healthcheck if no liveness/readiness was added. + if(!liveness_readiness_added && config_obj.isMember("Healthcheck")) + { + parse_healthcheck(config_obj["Healthcheck"], container); + } +} + +void docker_async_source::set_query_image_info(bool query_image_info) +{ + g_logger.format(sinsp_logger::SEV_DEBUG, + "docker_async: Setting query_image_info=%s", + (query_image_info ? "true" : "false")); + + m_query_image_info = query_image_info; +} + +void docker_async_source::parse_json_mounts(const Json::Value &mnt_obj, vector &mounts) +{ + if(!mnt_obj.isNull() && mnt_obj.isArray()) + { + for(uint32_t i=0; i imageDigestSet; + for(const auto& rdig : img_root["RepoDigests"]) + { + if(rdig.isString()) + { + string repodigest = rdig.asString(); + string digest = repodigest.substr(repodigest.find('@')+1); + imageDigestSet.insert(digest); + if(container.m_imagerepo.empty()) + { + container.m_imagerepo = repodigest.substr(0, repodigest.find('@')); + } + if(repodigest.find(container.m_imagerepo) != string::npos) + { + container.m_imagedigest = digest; + break; + } + } + } + for(const auto& rtag : img_root["RepoTags"]) + { + if(rtag.isString()) + { + string repotag = rtag.asString(); + if(container.m_imagerepo.empty()) + { + container.m_imagerepo = repotag.substr(0, repotag.rfind(":")); + } + if(repotag.find(container.m_imagerepo) != string::npos) + { + container.m_imagetag = repotag.substr(repotag.rfind(":")+1); + break; + } + } + } + // fix image digest for locally tagged images or multiple repo digests. + // Case 1: One repo digest with many tags. + // Case 2: Many repo digests with the same digest value. + if(container.m_imagedigest.empty() && imageDigestSet.size() == 1) { + container.m_imagedigest = *imageDigestSet.begin(); + } + } + else + { + g_logger.format(sinsp_logger::SEV_ERROR, + "docker_async (%s) image (%s): Could not parse json image info \"%s\"", + request.container_id.c_str(), + container.m_imageid.c_str(), + img_json.c_str()); + } + } + else + { + g_logger.format(sinsp_logger::SEV_ERROR, + "docker_async (%s) image (%s): Could not fetch image info", + request.container_id.c_str(), + container.m_imageid.c_str()); + } + + } + if(container.m_imagetag.empty()) + { + container.m_imagetag = "latest"; + } + + container.m_full_id = root["Id"].asString(); + container.m_name = root["Name"].asString(); + // k8s Docker container names could have '/' as the first character. + if(!container.m_name.empty() && container.m_name[0] == '/') + { + container.m_name = container.m_name.substr(1); + } + if(container.m_name.find("k8s_POD") == 0) + { + container.m_is_pod_sandbox = true; + } + + // Get the created time - this will be string format i.e. "%Y-%m-%dT%H:%M:%SZ" + // Convert it to seconds. This can be done with get_epoc_utc_seconds() + container.m_created_time = static_cast(get_epoch_utc_seconds(root["Created"].asString())); + + const Json::Value& net_obj = root["NetworkSettings"]; + + string ip = net_obj["IPAddress"].asString(); + + if(ip.empty()) + { + const Json::Value& hconfig_obj = root["HostConfig"]; + string net_mode = hconfig_obj["NetworkMode"].asString(); + + if(strncmp(net_mode.c_str(), "container:", strlen("container:")) == 0) + { + std::string secondary_container_id = net_mode.substr(net_mode.find(":") + 1); + + sinsp_container_info pcnt; + pcnt.m_id = secondary_container_id; + + // This is a *blocking* fetch of the + // secondary container, but we're in a + // separate thread so this is ok. + g_logger.format(sinsp_logger::SEV_DEBUG, + "docker_async (%s), secondary (%s): Doing blocking fetch of secondary container", + request.container_id.c_str(), + secondary_container_id.c_str()); + + if(parse_docker(docker_lookup_request(secondary_container_id, + request.docker_socket, + false /*don't request size since we just need the IP*/), + pcnt)) + { + g_logger.format(sinsp_logger::SEV_DEBUG, + "docker_async (%s), secondary (%s): Secondary fetch successful", + request.container_id.c_str(), + secondary_container_id.c_str()); + container.m_container_ip = pcnt.m_container_ip; + } + else + { + g_logger.format(sinsp_logger::SEV_ERROR, + "docker_async (%s), secondary (%s): Secondary fetch failed", + request.container_id.c_str(), + secondary_container_id.c_str()); + } + } + } + else + { + if(inet_pton(AF_INET, ip.c_str(), &container.m_container_ip) == -1) + { + ASSERT(false); + } + container.m_container_ip = ntohl(container.m_container_ip); + } + + vector ports = net_obj["Ports"].getMemberNames(); + for(vector::const_iterator it = ports.begin(); it != ports.end(); ++it) + { + size_t tcp_pos = it->find("/tcp"); + if(tcp_pos == string::npos) + { + continue; + } + + uint16_t container_port = atoi(it->c_str()); + + const Json::Value& v = net_obj["Ports"][*it]; + if(v.isArray()) + { + for(uint32_t j = 0; j < v.size(); ++j) + { + sinsp_container_info::container_port_mapping port_mapping; + + ip = v[j]["HostIp"].asString(); + string port = v[j]["HostPort"].asString(); + + if(inet_pton(AF_INET, ip.c_str(), &port_mapping.m_host_ip) == -1) + { + ASSERT(false); + continue; + } + port_mapping.m_host_ip = ntohl(port_mapping.m_host_ip); + + port_mapping.m_container_port = container_port; + port_mapping.m_host_port = atoi(port.c_str()); + container.m_port_mappings.push_back(port_mapping); + } + } + } + + vector labels = config_obj["Labels"].getMemberNames(); + for(vector::const_iterator it = labels.begin(); it != labels.end(); ++it) + { + string val = config_obj["Labels"][*it].asString(); + if(val.length() <= sinsp_container_info::m_container_label_max_length ) { + container.m_labels[*it] = val; + } + } + + const Json::Value& env_vars = config_obj["Env"]; + + for(const auto& env_var : env_vars) + { + if(env_var.isString()) + { + container.m_env.emplace_back(env_var.asString()); + } + } + + const auto& host_config_obj = root["HostConfig"]; + container.m_memory_limit = host_config_obj["Memory"].asInt64(); + container.m_swap_limit = host_config_obj["MemorySwap"].asInt64(); + const auto cpu_shares = host_config_obj["CpuShares"].asInt64(); + if(cpu_shares > 0) + { + container.m_cpu_shares = cpu_shares; + } + + /** + * 2 separate docker APIs use CFS CPU scheduler to constrain container CPU usage + * Reference: https://docs.docker.com/engine/reference/run/ + * 1) docker run --cpus= + * is converted into a cfs_cpu_quota value for the default cfs_cpu_period=100000 + * cfs_cpu_period cannot be changed with this API + * For example, if =0.5, cfs_cpu_quota=50000 and cfs_cpu_period=100000 + * 2) docker run --cpu-quota= --cpu-period= + * CFS quota and/or period can be set directly. The default period is 100000 and default quota + * is 0 (which translates to unconstrained) + * For example, if =12345 and =67890, then cfs_cpu_quota=12345 and cfs_cpu_period=67890 + * These 2 APIs are mutually exclusive: docker throws an error if an attempt is made to use --cpus in combination + * with either --cpu-quota or --cpu-period + * + * docker_response json output: + * 1) When --cpus is used, the value is returned as NanoCpus; both CpuQuota and CpuPeriod are 0 + * Since cfs_cpu_period=100000=10^5 and 10^9 NanoCpus is 1 CPU, which translates to cfs_cpu_quota=100000=10^5, + * we need to divide NanoCpus by 10^4=10000 to convert NanoCpus into cfs_cpu_quota + * + * 2) When --cpu-quota and/or --cpu-period are used, the corresponding values are returned; NanoCpus is 0 + */ + container.m_cpu_quota = max(host_config_obj["CpuQuota"].asInt64(), host_config_obj["NanoCpus"].asInt64()/10000); + const auto cpu_period = host_config_obj["CpuPeriod"].asInt64(); + if(cpu_period > 0) + { + container.m_cpu_period = cpu_period; + } + const auto cpuset_cpus = host_config_obj["CpusetCpus"].asString(); + if (!cpuset_cpus.empty()) + { + libsinsp::cgroup_list_counter counter; + container.m_cpuset_cpu_count = counter(cpuset_cpus.c_str()); + } + const Json::Value& privileged = host_config_obj["Privileged"]; + if(!privileged.isNull() && privileged.isBool()) + { + container.m_privileged = privileged.asBool(); + } + + parse_json_mounts(root["Mounts"], container.m_mounts); + + container.m_size_rw_bytes = root["SizeRw"].asInt64(); + +#ifdef HAS_ANALYZER + sinsp_utils::find_env(container.m_sysdig_agent_conf, container.get_env(), "SYSDIG_AGENT_CONF"); + // container.m_sysdig_agent_conf = get_docker_env(env_vars, "SYSDIG_AGENT_CONF"); +#endif + + g_logger.format(sinsp_logger::SEV_DEBUG, + "docker_async (%s): parse_docker returning true", + request.container_id.c_str()); + return true; +} + diff --git a/userspace/libsinsp/container_engine/docker/async_source.h b/userspace/libsinsp/container_engine/docker/async_source.h new file mode 100644 index 000000000..f9efbc8f5 --- /dev/null +++ b/userspace/libsinsp/container_engine/docker/async_source.h @@ -0,0 +1,72 @@ +#pragma once + +#include "async_key_value_source.h" +#include "container_info.h" + +#include "container_engine/docker/connection.h" +#include "container_engine/docker/lookup_request.h" + +namespace libsinsp { +namespace container_engine { + +class container_cache_interface; + +class docker_async_source : public sysdig::async_key_value_source +{ +public: + docker_async_source(uint64_t max_wait_ms, uint64_t ttl_ms, container_cache_interface *cache); + virtual ~docker_async_source(); + + static void parse_json_mounts(const Json::Value &mnt_obj, std::vector &mounts); + static void set_query_image_info(bool query_image_info); + +protected: + void run_impl(); + +private: + bool parse_docker(const docker_lookup_request& request, sinsp_container_info& container); + + // Look for a pod specification in this container's labels and + // if found set spec to the pod spec. + bool get_k8s_pod_spec(const Json::Value &config_obj, + Json::Value &spec); + + std::string normalize_arg(const std::string &arg); + + // Parse a healthcheck out of the provided healthcheck object, + // updating the container info with any healthcheck found. + void parse_healthcheck(const Json::Value &healthcheck_obj, + sinsp_container_info &container); + + // Parse either a readiness or liveness probe out of the + // provided object, updating the container info with any probe + // found. Returns true if the healthcheck/livenesss/readiness + // probe info was found and could be parsed. + bool parse_liveness_readiness_probe(const Json::Value &probe_obj, + sinsp_container_info::container_health_probe::probe_type ptype, + sinsp_container_info &container); + + // See if this config has a io.kubernetes.sandbox.id label + // referring to a different container. (NOTE: this is not the + // same as docker's sandbox id, which refers to networks.) If + // it does, try to copy the health checks from that container + // to the provided container_info pointer. Returns true if a + // sandbox container id was found, the corresponding container + // was found, and if the health checks could be copied from + // that container. + bool get_sandbox_liveness_readiness_probes(const Json::Value &config_obj, + sinsp_container_info &container); + + // Parse all healthchecks/liveness probes/readiness probes out + // of the provided object, updating the container info as required. + void parse_health_probes(const Json::Value &config_obj, + sinsp_container_info &container); + + container_cache_interface *m_cache; + docker_connection m_connection; + static bool m_query_image_info; +}; + + +} +} diff --git a/userspace/libsinsp/container_engine/docker_common.cpp b/userspace/libsinsp/container_engine/docker_common.cpp index fd9e788bc..0a17cdc48 100644 --- a/userspace/libsinsp/container_engine/docker_common.cpp +++ b/userspace/libsinsp/container_engine/docker_common.cpp @@ -27,370 +27,6 @@ limitations under the License. using namespace libsinsp::container_engine; -docker_async_source::docker_async_source(uint64_t max_wait_ms, - uint64_t ttl_ms, - container_cache_interface *cache) - : async_key_value_source(max_wait_ms, ttl_ms), - m_cache(cache) -{ -} - -docker_async_source::~docker_async_source() -{ - this->stop(); - g_logger.format(sinsp_logger::SEV_DEBUG, - "docker_async: Source destructor"); -} - -void docker_async_source::run_impl() -{ - docker_lookup_request request; - - while (dequeue_next_key(request)) - { - g_logger.format(sinsp_logger::SEV_DEBUG, - "docker_async (%s : %s): Source dequeued key", - request.container_id.c_str(), - request.request_rw_size ? "true" : "false"); - - sinsp_container_info res; - - res.m_lookup_state = sinsp_container_lookup_state::SUCCESSFUL; - res.m_type = CT_DOCKER; - res.m_id = request.container_id; - - if(!parse_docker(request, res)) - { - // This is not always an error e.g. when using - // containerd as the runtime. Since the cgroup - // names are often identical between - // containerd and docker, we have to try to - // fetch both. - g_logger.format(sinsp_logger::SEV_DEBUG, - "docker_async (%s): Failed to get Docker metadata, returning successful=false", - request.container_id.c_str()); - res.m_lookup_state = sinsp_container_lookup_state::FAILED; - } - - g_logger.format(sinsp_logger::SEV_DEBUG, - "docker_async (%s): Parse successful, storing value", - request.container_id.c_str()); - - // Return a result object either way, to ensure any - // new container callbacks are called. - store_value(request, res); - } -} - -bool docker_async_source::m_query_image_info = true; - -void docker::parse_json_mounts(const Json::Value &mnt_obj, vector &mounts) -{ - if(!mnt_obj.isNull() && mnt_obj.isArray()) - { - for(uint32_t i=0; i args; - - for(uint32_t i = 2; i < test_obj.size(); i++) - { - args.push_back(normalize_arg(test_obj[i].asString())); - } - - g_logger.format(sinsp_logger::SEV_DEBUG, - "docker (%s): Setting PT_HEALTHCHECK exe=%s nargs=%d", - container.m_id.c_str(), exe.c_str(), args.size()); - - container.m_health_probes.emplace_back(sinsp_container_info::container_health_probe::PT_HEALTHCHECK, - std::move(exe), - std::move(args)); - } - else if(test_obj[0].asString() == "CMD-SHELL") - { - std::string exe = "/bin/sh"; - std::vector args; - - args.push_back("-c"); - args.push_back(test_obj[1].asString()); - - g_logger.format(sinsp_logger::SEV_DEBUG, - "docker (%s): Setting PT_HEALTHCHECK exe=%s nargs=%d", - container.m_id.c_str(), exe.c_str(), args.size()); - - container.m_health_probes.emplace_back(sinsp_container_info::container_health_probe::PT_HEALTHCHECK, - std::move(exe), - std::move(args)); - } - else - { - g_logger.format(sinsp_logger::SEV_WARNING, "Could not parse health check from %s (Expected CMD/CMD-SHELL for multi-element Test array)", - Json::FastWriter().write(healthcheck_obj).c_str()); - return; - } -} - -bool docker_async_source::parse_liveness_readiness_probe(const Json::Value &probe_obj, - sinsp_container_info::container_health_probe::probe_type ptype, - sinsp_container_info &container) -{ - if(probe_obj.isNull() || - !probe_obj.isMember("exec") || - !probe_obj["exec"].isMember("command")) - { - g_logger.format(sinsp_logger::SEV_WARNING, "Could not parse liveness/readiness probe from %s", - Json::FastWriter().write(probe_obj).c_str()); - return false; - } - - const Json::Value command_obj = probe_obj["exec"]["command"]; - - if(!command_obj.isNull() && command_obj.isArray()) - { - std::string exe; - std::vector args; - - exe = normalize_arg(command_obj[0].asString()); - for(uint32_t i = 1; i < command_obj.size(); i++) - { - args.push_back(normalize_arg(command_obj[i].asString())); - } - - g_logger.format(sinsp_logger::SEV_DEBUG, - "docker (%s): Setting %s exe=%s nargs=%d", - container.m_id.c_str(), - sinsp_container_info::container_health_probe::probe_type_names[ptype].c_str(), - exe.c_str(), args.size()); - - container.m_health_probes.emplace_back(ptype, std::move(exe), std::move(args)); - } - - return true; -} - -bool docker_async_source::get_sandbox_liveness_readiness_probes(const Json::Value &config_obj, - sinsp_container_info &container) -{ - std::string sandbox_container_id; - std::string sandbox_label = "io.kubernetes.sandbox.id"; - - if(config_obj.isNull() || - !config_obj.isMember("Labels") || - !config_obj["Labels"].isMember(sandbox_label)) - { - SINSP_DEBUG("docker (%s): No sandbox label found, not copying liveness/readiness probes", - container.m_id.c_str()); - return false; - } - - sandbox_container_id = config_obj["Labels"][sandbox_label].asString(); - - if(sandbox_container_id.size() > 12) - { - sandbox_container_id.resize(12); - } - - sinsp_container_info::ptr_t sandbox_container = m_cache->get_container(sandbox_container_id); - - if(!sandbox_container) - { - SINSP_DEBUG("docker (%s): Sandbox container %s doesn't exist, not copying liveness/readiness probes", - container.m_id.c_str(), sandbox_container_id.c_str()); - return false; - } - - if(sandbox_container->m_health_probes.size() == 0) - { - SINSP_DEBUG("docker (%s): Sandbox container %s has no liveness/readiness probes, not copying", - container.m_id.c_str(), sandbox_container_id.c_str()); - return false; - } - - SINSP_DEBUG("docker (%s): Copying liveness/readiness probes from sandbox container %s", - container.m_id.c_str(), sandbox_container_id.c_str()); - container.m_health_probes = sandbox_container->m_health_probes; - - return true; -} - -void docker_async_source::parse_health_probes(const Json::Value &config_obj, - sinsp_container_info &container) -{ - Json::Value spec; - bool liveness_readiness_added = false; - - // When parsing the full container json for live containers, a label contains stringified json that - // contains the probes. - if (get_k8s_pod_spec(config_obj, spec)) - { - g_logger.format(sinsp_logger::SEV_DEBUG, - "docker (%s): Parsing liveness/readiness probes from pod spec", - container.m_id.c_str()); - - if(spec.isMember("livenessProbe")) - { - if(parse_liveness_readiness_probe(spec["livenessProbe"], - sinsp_container_info::container_health_probe::PT_LIVENESS_PROBE, - container)) - { - liveness_readiness_added = true; - } - } - else if(spec.isMember("readinessProbe")) - { - if(parse_liveness_readiness_probe(spec["readinessProbe"], - sinsp_container_info::container_health_probe::PT_READINESS_PROBE, - container)) - { - liveness_readiness_added = true; - } - } - } - // Otherwise, try to copy the liveness/readiness probe from the sandbox container, if it exists. - else if (get_sandbox_liveness_readiness_probes(config_obj, container)) - { - liveness_readiness_added = true; - } - else - { - g_logger.format(sinsp_logger::SEV_DEBUG, - "docker (%s): No liveness/readiness probes found", - container.m_id.c_str()); - } - - // To avoid any confusion about containers that both refer to - // a healthcheck and liveness/readiness probe, we only - // consider a healthcheck if no liveness/readiness was added. - if(!liveness_readiness_added && config_obj.isMember("Healthcheck")) - { - parse_healthcheck(config_obj["Healthcheck"], container); - } -} - -void docker_async_source::set_query_image_info(bool query_image_info) -{ - g_logger.format(sinsp_logger::SEV_DEBUG, - "docker_async: Setting query_image_info=%s", - (query_image_info ? "true" : "false")); - - m_query_image_info = query_image_info; -} std::string docker::s_incomplete_info_name = "incomplete"; @@ -476,381 +112,6 @@ void docker::parse_docker_async(const string& container_id, container_cache_inte } } -bool docker_async_source::parse_docker(const docker_lookup_request& request, sinsp_container_info& container) -{ - string json; - - g_logger.format(sinsp_logger::SEV_DEBUG, - "docker_async (%s): Looking up info for container", - request.container_id.c_str()); - - std::string api_request = "/containers/" + request.container_id + "/json"; - if(request.request_rw_size) - { - api_request += "?size=true"; - } - - docker_connection::docker_response resp = m_connection.get_docker(request, api_request, json); - - switch(resp) { - case docker_connection::docker_response::RESP_BAD_REQUEST: - g_logger.format(sinsp_logger::SEV_DEBUG, - "docker_async (%s): Initial url fetch failed, trying w/o api version", - request.container_id.c_str()); - - m_connection.set_api_version(""); - json = ""; - resp = m_connection.get_docker(request, "/containers/" + request.container_id + "/json", json); - if (resp == docker_connection::docker_response::RESP_OK) - { - break; - } - /* FALLTHRU */ - case docker_connection::docker_response::RESP_ERROR: - g_logger.format(sinsp_logger::SEV_DEBUG, - "docker_async (%s): Url fetch failed, returning false", - request.container_id.c_str()); - - return false; - - case docker_connection::docker_response::RESP_OK: - break; - } - - g_logger.format(sinsp_logger::SEV_DEBUG, - "docker_async (%s): Parsing containers response \"%s\"", - request.container_id.c_str(), - json.c_str()); - - Json::Value root; - Json::Reader reader; - bool parsingSuccessful = reader.parse(json, root); - if(!parsingSuccessful) - { - g_logger.format(sinsp_logger::SEV_ERROR, - "docker_async (%s): Could not parse json \"%s\", returning false", - request.container_id.c_str(), - json.c_str()); - - ASSERT(false); - return false; - } - - const Json::Value& config_obj = root["Config"]; - - container.m_image = config_obj["Image"].asString(); - - string imgstr = root["Image"].asString(); - size_t cpos = imgstr.find(":"); - if(cpos != string::npos) - { - container.m_imageid = imgstr.substr(cpos + 1); - } - - parse_health_probes(config_obj, container); - - // containers can be spawned using just the imageID as image name, - // with or without the hash prefix (e.g. sha256:) - bool no_name = !container.m_imageid.empty() && - strncmp(container.m_image.c_str(), container.m_imageid.c_str(), - MIN(container.m_image.length(), container.m_imageid.length())) == 0; - no_name |= !imgstr.empty() && - strncmp(container.m_image.c_str(), imgstr.c_str(), - MIN(container.m_image.length(), imgstr.length())) == 0; - - if(!no_name || !m_query_image_info) - { - string hostname, port; - sinsp_utils::split_container_image(container.m_image, - hostname, - port, - container.m_imagerepo, - container.m_imagetag, - container.m_imagedigest, - false); - } - - if(m_query_image_info && !container.m_imageid.empty() && - (no_name || container.m_imagedigest.empty() || (!container.m_imagedigest.empty() && container.m_imagetag.empty()))) - { - g_logger.format(sinsp_logger::SEV_DEBUG, - "docker_async (%s) image (%s): Fetching image info", - request.container_id.c_str(), - container.m_imageid.c_str()); - - string img_json; - std::string url = "/images/" + container.m_imageid + "/json?digests=1"; - - g_logger.format(sinsp_logger::SEV_DEBUG, - "docker_async url: %s", - url.c_str()); - - if(m_connection.get_docker(request, url, img_json) == docker_connection::docker_response::RESP_OK) - { - g_logger.format(sinsp_logger::SEV_DEBUG, - "docker_async (%s) image (%s): Image info fetch returned \"%s\"", - request.container_id.c_str(), - container.m_imageid.c_str(), - img_json.c_str()); - - Json::Value img_root; - if(reader.parse(img_json, img_root)) - { - // img_root["RepoDigests"] contains only digests for images pulled from registries. - // If an image gets retagged and is never pushed to any registry, we will not find - // that entry in container.m_imagerepo. Also, for locally built images we have the - // same issue. This leads to container.m_imagedigest being empty as well. - unordered_set imageDigestSet; - for(const auto& rdig : img_root["RepoDigests"]) - { - if(rdig.isString()) - { - string repodigest = rdig.asString(); - string digest = repodigest.substr(repodigest.find('@')+1); - imageDigestSet.insert(digest); - if(container.m_imagerepo.empty()) - { - container.m_imagerepo = repodigest.substr(0, repodigest.find('@')); - } - if(repodigest.find(container.m_imagerepo) != string::npos) - { - container.m_imagedigest = digest; - break; - } - } - } - for(const auto& rtag : img_root["RepoTags"]) - { - if(rtag.isString()) - { - string repotag = rtag.asString(); - if(container.m_imagerepo.empty()) - { - container.m_imagerepo = repotag.substr(0, repotag.rfind(":")); - } - if(repotag.find(container.m_imagerepo) != string::npos) - { - container.m_imagetag = repotag.substr(repotag.rfind(":")+1); - break; - } - } - } - // fix image digest for locally tagged images or multiple repo digests. - // Case 1: One repo digest with many tags. - // Case 2: Many repo digests with the same digest value. - if(container.m_imagedigest.empty() && imageDigestSet.size() == 1) { - container.m_imagedigest = *imageDigestSet.begin(); - } - } - else - { - g_logger.format(sinsp_logger::SEV_ERROR, - "docker_async (%s) image (%s): Could not parse json image info \"%s\"", - request.container_id.c_str(), - container.m_imageid.c_str(), - img_json.c_str()); - } - } - else - { - g_logger.format(sinsp_logger::SEV_ERROR, - "docker_async (%s) image (%s): Could not fetch image info", - request.container_id.c_str(), - container.m_imageid.c_str()); - } - - } - if(container.m_imagetag.empty()) - { - container.m_imagetag = "latest"; - } - - container.m_full_id = root["Id"].asString(); - container.m_name = root["Name"].asString(); - // k8s Docker container names could have '/' as the first character. - if(!container.m_name.empty() && container.m_name[0] == '/') - { - container.m_name = container.m_name.substr(1); - } - if(container.m_name.find("k8s_POD") == 0) - { - container.m_is_pod_sandbox = true; - } - - // Get the created time - this will be string format i.e. "%Y-%m-%dT%H:%M:%SZ" - // Convert it to seconds. This can be done with get_epoc_utc_seconds() - container.m_created_time = static_cast(get_epoch_utc_seconds(root["Created"].asString())); - - const Json::Value& net_obj = root["NetworkSettings"]; - - string ip = net_obj["IPAddress"].asString(); - - if(ip.empty()) - { - const Json::Value& hconfig_obj = root["HostConfig"]; - string net_mode = hconfig_obj["NetworkMode"].asString(); - - if(strncmp(net_mode.c_str(), "container:", strlen("container:")) == 0) - { - std::string secondary_container_id = net_mode.substr(net_mode.find(":") + 1); - - sinsp_container_info pcnt; - pcnt.m_id = secondary_container_id; - - // This is a *blocking* fetch of the - // secondary container, but we're in a - // separate thread so this is ok. - g_logger.format(sinsp_logger::SEV_DEBUG, - "docker_async (%s), secondary (%s): Doing blocking fetch of secondary container", - request.container_id.c_str(), - secondary_container_id.c_str()); - - if(parse_docker(docker_lookup_request(secondary_container_id, - request.docker_socket, - false /*don't request size since we just need the IP*/), - pcnt)) - { - g_logger.format(sinsp_logger::SEV_DEBUG, - "docker_async (%s), secondary (%s): Secondary fetch successful", - request.container_id.c_str(), - secondary_container_id.c_str()); - container.m_container_ip = pcnt.m_container_ip; - } - else - { - g_logger.format(sinsp_logger::SEV_ERROR, - "docker_async (%s), secondary (%s): Secondary fetch failed", - request.container_id.c_str(), - secondary_container_id.c_str()); - } - } - } - else - { - if(inet_pton(AF_INET, ip.c_str(), &container.m_container_ip) == -1) - { - ASSERT(false); - } - container.m_container_ip = ntohl(container.m_container_ip); - } - - vector ports = net_obj["Ports"].getMemberNames(); - for(vector::const_iterator it = ports.begin(); it != ports.end(); ++it) - { - size_t tcp_pos = it->find("/tcp"); - if(tcp_pos == string::npos) - { - continue; - } - - uint16_t container_port = atoi(it->c_str()); - - const Json::Value& v = net_obj["Ports"][*it]; - if(v.isArray()) - { - for(uint32_t j = 0; j < v.size(); ++j) - { - sinsp_container_info::container_port_mapping port_mapping; - - ip = v[j]["HostIp"].asString(); - string port = v[j]["HostPort"].asString(); - - if(inet_pton(AF_INET, ip.c_str(), &port_mapping.m_host_ip) == -1) - { - ASSERT(false); - continue; - } - port_mapping.m_host_ip = ntohl(port_mapping.m_host_ip); - - port_mapping.m_container_port = container_port; - port_mapping.m_host_port = atoi(port.c_str()); - container.m_port_mappings.push_back(port_mapping); - } - } - } - - vector labels = config_obj["Labels"].getMemberNames(); - for(vector::const_iterator it = labels.begin(); it != labels.end(); ++it) - { - string val = config_obj["Labels"][*it].asString(); - if(val.length() <= sinsp_container_info::m_container_label_max_length ) { - container.m_labels[*it] = val; - } - } - - const Json::Value& env_vars = config_obj["Env"]; - - for(const auto& env_var : env_vars) - { - if(env_var.isString()) - { - container.m_env.emplace_back(env_var.asString()); - } - } - - const auto& host_config_obj = root["HostConfig"]; - container.m_memory_limit = host_config_obj["Memory"].asInt64(); - container.m_swap_limit = host_config_obj["MemorySwap"].asInt64(); - const auto cpu_shares = host_config_obj["CpuShares"].asInt64(); - if(cpu_shares > 0) - { - container.m_cpu_shares = cpu_shares; - } - - /** - * 2 separate docker APIs use CFS CPU scheduler to constrain container CPU usage - * Reference: https://docs.docker.com/engine/reference/run/ - * 1) docker run --cpus= - * is converted into a cfs_cpu_quota value for the default cfs_cpu_period=100000 - * cfs_cpu_period cannot be changed with this API - * For example, if =0.5, cfs_cpu_quota=50000 and cfs_cpu_period=100000 - * 2) docker run --cpu-quota= --cpu-period= - * CFS quota and/or period can be set directly. The default period is 100000 and default quota - * is 0 (which translates to unconstrained) - * For example, if =12345 and =67890, then cfs_cpu_quota=12345 and cfs_cpu_period=67890 - * These 2 APIs are mutually exclusive: docker throws an error if an attempt is made to use --cpus in combination - * with either --cpu-quota or --cpu-period - * - * docker_response json output: - * 1) When --cpus is used, the value is returned as NanoCpus; both CpuQuota and CpuPeriod are 0 - * Since cfs_cpu_period=100000=10^5 and 10^9 NanoCpus is 1 CPU, which translates to cfs_cpu_quota=100000=10^5, - * we need to divide NanoCpus by 10^4=10000 to convert NanoCpus into cfs_cpu_quota - * - * 2) When --cpu-quota and/or --cpu-period are used, the corresponding values are returned; NanoCpus is 0 - */ - container.m_cpu_quota = max(host_config_obj["CpuQuota"].asInt64(), host_config_obj["NanoCpus"].asInt64()/10000); - const auto cpu_period = host_config_obj["CpuPeriod"].asInt64(); - if(cpu_period > 0) - { - container.m_cpu_period = cpu_period; - } - const auto cpuset_cpus = host_config_obj["CpusetCpus"].asString(); - if (!cpuset_cpus.empty()) - { - libsinsp::cgroup_list_counter counter; - container.m_cpuset_cpu_count = counter(cpuset_cpus.c_str()); - } - const Json::Value& privileged = host_config_obj["Privileged"]; - if(!privileged.isNull() && privileged.isBool()) - { - container.m_privileged = privileged.asBool(); - } - - docker::parse_json_mounts(root["Mounts"], container.m_mounts); - - container.m_size_rw_bytes = root["SizeRw"].asInt64(); - -#ifdef HAS_ANALYZER - sinsp_utils::find_env(container.m_sysdig_agent_conf, container.get_env(), "SYSDIG_AGENT_CONF"); - // container.m_sysdig_agent_conf = get_docker_env(env_vars, "SYSDIG_AGENT_CONF"); -#endif - - g_logger.format(sinsp_logger::SEV_DEBUG, - "docker_async (%s): parse_docker returning true", - request.container_id.c_str()); - return true; -} - void docker::update_with_size(const std::string &container_id) { auto cb = [this](const docker_lookup_request& request, const sinsp_container_info& res) { diff --git a/userspace/libsinsp/parsers.cpp b/userspace/libsinsp/parsers.cpp index 00be8b04d..566335bac 100644 --- a/userspace/libsinsp/parsers.cpp +++ b/userspace/libsinsp/parsers.cpp @@ -4741,7 +4741,7 @@ void sinsp_parser::parse_container_json_evt(sinsp_evt *evt) } #if !defined(MINIMAL_BUILD) && !defined(_WIN32) - libsinsp::container_engine::docker::parse_json_mounts(container["Mounts"], container_info->m_mounts); + libsinsp::container_engine::docker_async_source::parse_json_mounts(container["Mounts"], container_info->m_mounts); #endif sinsp_container_info::container_health_probe::parse_health_probes(container, container_info->m_health_probes); From 59873aa4ab620c69aa6929f22d38c46e1c70f1c1 Mon Sep 17 00:00:00 2001 From: Grzegorz Nosek Date: Fri, 16 Jul 2021 17:14:46 +0200 Subject: [PATCH 076/148] Pass a complete request to parse_docker_async We need full control over the request to make this method suitable for putting in a base class. Signed-off-by: Grzegorz Nosek --- userspace/libsinsp/container_engine/docker.h | 2 +- userspace/libsinsp/container_engine/docker_common.cpp | 9 +++++---- 2 files changed, 6 insertions(+), 5 deletions(-) diff --git a/userspace/libsinsp/container_engine/docker.h b/userspace/libsinsp/container_engine/docker.h index 99a4662d0..d1dcf1c52 100644 --- a/userspace/libsinsp/container_engine/docker.h +++ b/userspace/libsinsp/container_engine/docker.h @@ -72,7 +72,7 @@ class docker : public container_engine_base #endif protected: - void parse_docker_async(const std::string& container_id, container_cache_interface *cache); + void parse_docker_async(const docker_lookup_request& request, container_cache_interface *cache); std::unique_ptr m_docker_info_source; diff --git a/userspace/libsinsp/container_engine/docker_common.cpp b/userspace/libsinsp/container_engine/docker_common.cpp index 0a17cdc48..fce5341ac 100644 --- a/userspace/libsinsp/container_engine/docker_common.cpp +++ b/userspace/libsinsp/container_engine/docker_common.cpp @@ -40,6 +40,8 @@ bool docker::resolve(sinsp_threadinfo *tinfo, bool query_os_for_missing_info) return false; } + docker_lookup_request request(container_id, m_docker_sock, false /*don't request size*/); + if(!m_docker_info_source) { g_logger.log("docker_async: Creating docker async source", @@ -73,7 +75,7 @@ bool docker::resolve(sinsp_threadinfo *tinfo, bool query_os_for_missing_info) // give docker a chance to return metadata for this container cache->set_lookup_status(container_id, CT_DOCKER, sinsp_container_lookup_state::STARTED); - parse_docker_async(container_id, cache); + parse_docker_async(request, cache); } #endif return false; @@ -85,7 +87,7 @@ bool docker::resolve(sinsp_threadinfo *tinfo, bool query_os_for_missing_info) return container_info->is_successful(); } -void docker::parse_docker_async(const string& container_id, container_cache_interface *cache) +void docker::parse_docker_async(const docker_lookup_request& request, container_cache_interface *cache) { auto cb = [cache](const docker_lookup_request& request, const sinsp_container_info& res) { @@ -99,7 +101,6 @@ void docker::parse_docker_async(const string& container_id, container_cache_inte sinsp_container_info result; - docker_lookup_request request(container_id, m_docker_sock, false /*don't request size*/); if(m_docker_info_source->lookup(request, result, cb)) { // if a previous lookup call already found the metadata, process it now @@ -108,7 +109,7 @@ void docker::parse_docker_async(const string& container_id, container_cache_inte // This should *never* happen, as ttl is 0 (never wait) g_logger.format(sinsp_logger::SEV_ERROR, "docker_async (%s): Unexpected immediate return from docker_info_source.lookup()", - container_id.c_str()); + request.container_id.c_str()); } } From dd1088c38e114570d9cd700eec408ff64c1d19fe Mon Sep 17 00:00:00 2001 From: Grzegorz Nosek Date: Fri, 16 Jul 2021 17:25:10 +0200 Subject: [PATCH 077/148] Move most of docker::resolve to a new method All the code after the initial docker container detection is platform-agnostic and can be moved to a base class. Signed-off-by: Grzegorz Nosek --- userspace/libsinsp/container_engine/docker.h | 3 +++ .../container_engine/docker_common.cpp | 21 ++++++++++++------- 2 files changed, 17 insertions(+), 7 deletions(-) diff --git a/userspace/libsinsp/container_engine/docker.h b/userspace/libsinsp/container_engine/docker.h index d1dcf1c52..a4d2b6d81 100644 --- a/userspace/libsinsp/container_engine/docker.h +++ b/userspace/libsinsp/container_engine/docker.h @@ -87,6 +87,9 @@ class docker : public container_engine_base // implement container_engine_base bool resolve(sinsp_threadinfo *tinfo, bool query_os_for_missing_info) override; void update_with_size(const std::string& container_id) override; + + bool + resolve_impl(sinsp_threadinfo *tinfo, const docker_lookup_request& request, bool query_os_for_missing_info); }; } } diff --git a/userspace/libsinsp/container_engine/docker_common.cpp b/userspace/libsinsp/container_engine/docker_common.cpp index fce5341ac..2e0af4373 100644 --- a/userspace/libsinsp/container_engine/docker_common.cpp +++ b/userspace/libsinsp/container_engine/docker_common.cpp @@ -33,7 +33,6 @@ std::string docker::s_incomplete_info_name = "incomplete"; bool docker::resolve(sinsp_threadinfo *tinfo, bool query_os_for_missing_info) { std::string container_id, container_name; - container_cache_interface *cache = &container_cache(); if(!detect_docker(tinfo, container_id, container_name)) { @@ -42,6 +41,14 @@ bool docker::resolve(sinsp_threadinfo *tinfo, bool query_os_for_missing_info) docker_lookup_request request(container_id, m_docker_sock, false /*don't request size*/); + return resolve_impl(tinfo, request, query_os_for_missing_info); + +} + +bool +docker::resolve_impl(sinsp_threadinfo *tinfo, const docker_lookup_request& request, bool query_os_for_missing_info) +{ + container_cache_interface *cache = &container_cache(); if(!m_docker_info_source) { g_logger.log("docker_async: Creating docker async source", @@ -51,9 +58,9 @@ bool docker::resolve(sinsp_threadinfo *tinfo, bool query_os_for_missing_info) m_docker_info_source.reset(src); } - tinfo->m_container_id = container_id; + tinfo->m_container_id = request.container_id; - sinsp_container_info::ptr_t container_info = cache->get_container(container_id); + sinsp_container_info::ptr_t container_info = cache->get_container(request.container_id); if(!container_info) { @@ -61,20 +68,20 @@ bool docker::resolve(sinsp_threadinfo *tinfo, bool query_os_for_missing_info) { auto container = std::make_shared(); container->m_type = CT_DOCKER; - container->m_id = container_id; + container->m_id = request.container_id; cache->notify_new_container(*container); return true; } #ifdef HAS_CAPTURE - if(cache->should_lookup(container_id, CT_DOCKER)) + if(cache->should_lookup(request.container_id, CT_DOCKER)) { g_logger.format(sinsp_logger::SEV_DEBUG, "docker_async (%s): No existing container info", - container_id.c_str()); + request.container_id.c_str()); // give docker a chance to return metadata for this container - cache->set_lookup_status(container_id, CT_DOCKER, sinsp_container_lookup_state::STARTED); + cache->set_lookup_status(request.container_id, CT_DOCKER, sinsp_container_lookup_state::STARTED); parse_docker_async(request, cache); } #endif From 8cc4e603ed3ada7607bb5b5af9148feaab308fb2 Mon Sep 17 00:00:00 2001 From: Grzegorz Nosek Date: Fri, 16 Jul 2021 18:24:02 +0200 Subject: [PATCH 078/148] Move common docker bits to a base class Now we're ready to make clean OS-specific subclasses. Signed-off-by: Grzegorz Nosek --- userspace/libsinsp/CMakeLists.txt | 3 +- userspace/libsinsp/container_engine/docker.h | 13 +-- .../libsinsp/container_engine/docker/base.cpp | 86 +++++++++++++++++++ .../libsinsp/container_engine/docker/base.h | 31 +++++++ .../container_engine/docker_common.cpp | 75 ---------------- .../container_engine/docker_linux.cpp | 5 -- .../libsinsp/container_engine/docker_win.cpp | 5 -- 7 files changed, 122 insertions(+), 96 deletions(-) create mode 100644 userspace/libsinsp/container_engine/docker/base.cpp create mode 100644 userspace/libsinsp/container_engine/docker/base.h diff --git a/userspace/libsinsp/CMakeLists.txt b/userspace/libsinsp/CMakeLists.txt index 87be1bf56..0007e4e6a 100644 --- a/userspace/libsinsp/CMakeLists.txt +++ b/userspace/libsinsp/CMakeLists.txt @@ -140,7 +140,8 @@ if(NOT MINIMAL_BUILD) mesos_state.cpp sinsp_curl.cpp container_engine/docker_common.cpp - container_engine/docker/async_source.cpp) + container_engine/docker/async_source.cpp + container_engine/docker/base.cpp) if(WIN32) list(APPEND SINSP_SOURCES container_engine/docker_win.cpp diff --git a/userspace/libsinsp/container_engine/docker.h b/userspace/libsinsp/container_engine/docker.h index a4d2b6d81..734fc98e5 100644 --- a/userspace/libsinsp/container_engine/docker.h +++ b/userspace/libsinsp/container_engine/docker.h @@ -38,6 +38,7 @@ limitations under the License. #include "container_info.h" #include "container_engine/docker/async_source.h" +#include "container_engine/docker/base.h" #include "container_engine/docker/connection.h" #include "container_engine/docker/lookup_request.h" #include "container_engine/container_engine_base.h" @@ -50,17 +51,16 @@ class sinsp_threadinfo; namespace libsinsp { namespace container_engine { -class docker : public container_engine_base +class docker : public docker_base { public: #ifdef _WIN32 docker(container_cache_interface &cache, const wmi_handle_source&); #else - docker(container_cache_interface &cache) : container_engine_base(cache) + docker(container_cache_interface &cache) : docker_base(cache) {} #endif - void cleanup() override; // Container name only set for windows. For linux name must be fetched via lookup static bool detect_docker(const sinsp_threadinfo* tinfo, std::string& container_id, std::string &container_name); @@ -72,10 +72,6 @@ class docker : public container_engine_base #endif protected: - void parse_docker_async(const docker_lookup_request& request, container_cache_interface *cache); - - std::unique_ptr m_docker_info_source; - static std::string s_incomplete_info_name; #ifdef _WIN32 const wmi_handle_source& m_wmi_handle_source; @@ -87,9 +83,6 @@ class docker : public container_engine_base // implement container_engine_base bool resolve(sinsp_threadinfo *tinfo, bool query_os_for_missing_info) override; void update_with_size(const std::string& container_id) override; - - bool - resolve_impl(sinsp_threadinfo *tinfo, const docker_lookup_request& request, bool query_os_for_missing_info); }; } } diff --git a/userspace/libsinsp/container_engine/docker/base.cpp b/userspace/libsinsp/container_engine/docker/base.cpp new file mode 100644 index 000000000..bf9f129f0 --- /dev/null +++ b/userspace/libsinsp/container_engine/docker/base.cpp @@ -0,0 +1,86 @@ +#include "base.h" + +#include "sinsp.h" + +using namespace libsinsp::container_engine; + +void docker_base::cleanup() +{ + m_docker_info_source.reset(NULL); +} + +bool +docker_base::resolve_impl(sinsp_threadinfo *tinfo, const docker_lookup_request& request, bool query_os_for_missing_info) +{ + container_cache_interface *cache = &container_cache(); + if(!m_docker_info_source) + { + g_logger.log("docker_async: Creating docker async source", + sinsp_logger::SEV_DEBUG); + uint64_t max_wait_ms = 10000; + docker_async_source *src = new docker_async_source(docker_async_source::NO_WAIT_LOOKUP, max_wait_ms, cache); + m_docker_info_source.reset(src); + } + + tinfo->m_container_id = request.container_id; + + sinsp_container_info::ptr_t container_info = cache->get_container(request.container_id); + + if(!container_info) + { + if(!query_os_for_missing_info) + { + auto container = std::make_shared(); + container->m_type = CT_DOCKER; + container->m_id = request.container_id; + cache->notify_new_container(*container); + return true; + } + +#ifdef HAS_CAPTURE + if(cache->should_lookup(request.container_id, CT_DOCKER)) + { + g_logger.format(sinsp_logger::SEV_DEBUG, + "docker_async (%s): No existing container info", + request.container_id.c_str()); + + // give docker a chance to return metadata for this container + cache->set_lookup_status(request.container_id, CT_DOCKER, sinsp_container_lookup_state::STARTED); + parse_docker_async(request, cache); + } +#endif + return false; + } + + // Returning true will prevent other container engines from + // trying to resolve the container, so only return true if we + // have complete metadata. + return container_info->is_successful(); +} + +void docker_base::parse_docker_async(const docker_lookup_request& request, container_cache_interface *cache) +{ + auto cb = [cache](const docker_lookup_request& request, const sinsp_container_info& res) + { + g_logger.format(sinsp_logger::SEV_DEBUG, + "docker_async (%s): Source callback result=%d", + request.container_id.c_str(), + res.m_lookup_state); + + cache->notify_new_container(res); + }; + + sinsp_container_info result; + + if(m_docker_info_source->lookup(request, result, cb)) + { + // if a previous lookup call already found the metadata, process it now + cb(request, result); + + // This should *never* happen, as ttl is 0 (never wait) + g_logger.format(sinsp_logger::SEV_ERROR, + "docker_async (%s): Unexpected immediate return from docker_info_source.lookup()", + request.container_id.c_str()); + } +} + diff --git a/userspace/libsinsp/container_engine/docker/base.h b/userspace/libsinsp/container_engine/docker/base.h new file mode 100644 index 000000000..99c1c8f54 --- /dev/null +++ b/userspace/libsinsp/container_engine/docker/base.h @@ -0,0 +1,31 @@ +#pragma once + +#include "container_engine/container_engine_base.h" +#include "container_engine/docker/async_source.h" + +class sinsp_threadinfo; + +namespace libsinsp { +namespace container_engine { + +class docker_lookup_request; + +class docker_base : public container_engine_base +{ +public: + docker_base(container_cache_interface &cache) : container_engine_base(cache) + {} + + void cleanup() override; + +protected: + void parse_docker_async(const docker_lookup_request& request, container_cache_interface *cache); + + bool resolve_impl(sinsp_threadinfo *tinfo, const docker_lookup_request& request, + bool query_os_for_missing_info); + + std::unique_ptr m_docker_info_source; +}; + +} +} diff --git a/userspace/libsinsp/container_engine/docker_common.cpp b/userspace/libsinsp/container_engine/docker_common.cpp index 2e0af4373..b418dd65d 100644 --- a/userspace/libsinsp/container_engine/docker_common.cpp +++ b/userspace/libsinsp/container_engine/docker_common.cpp @@ -45,81 +45,6 @@ bool docker::resolve(sinsp_threadinfo *tinfo, bool query_os_for_missing_info) } -bool -docker::resolve_impl(sinsp_threadinfo *tinfo, const docker_lookup_request& request, bool query_os_for_missing_info) -{ - container_cache_interface *cache = &container_cache(); - if(!m_docker_info_source) - { - g_logger.log("docker_async: Creating docker async source", - sinsp_logger::SEV_DEBUG); - uint64_t max_wait_ms = 10000; - docker_async_source *src = new docker_async_source(docker_async_source::NO_WAIT_LOOKUP, max_wait_ms, cache); - m_docker_info_source.reset(src); - } - - tinfo->m_container_id = request.container_id; - - sinsp_container_info::ptr_t container_info = cache->get_container(request.container_id); - - if(!container_info) - { - if(!query_os_for_missing_info) - { - auto container = std::make_shared(); - container->m_type = CT_DOCKER; - container->m_id = request.container_id; - cache->notify_new_container(*container); - return true; - } - -#ifdef HAS_CAPTURE - if(cache->should_lookup(request.container_id, CT_DOCKER)) - { - g_logger.format(sinsp_logger::SEV_DEBUG, - "docker_async (%s): No existing container info", - request.container_id.c_str()); - - // give docker a chance to return metadata for this container - cache->set_lookup_status(request.container_id, CT_DOCKER, sinsp_container_lookup_state::STARTED); - parse_docker_async(request, cache); - } -#endif - return false; - } - - // Returning true will prevent other container engines from - // trying to resolve the container, so only return true if we - // have complete metadata. - return container_info->is_successful(); -} - -void docker::parse_docker_async(const docker_lookup_request& request, container_cache_interface *cache) -{ - auto cb = [cache](const docker_lookup_request& request, const sinsp_container_info& res) - { - g_logger.format(sinsp_logger::SEV_DEBUG, - "docker_async (%s): Source callback result=%d", - request.container_id.c_str(), - res.m_lookup_state); - - cache->notify_new_container(res); - }; - - sinsp_container_info result; - - if(m_docker_info_source->lookup(request, result, cb)) - { - // if a previous lookup call already found the metadata, process it now - cb(request, result); - - // This should *never* happen, as ttl is 0 (never wait) - g_logger.format(sinsp_logger::SEV_ERROR, - "docker_async (%s): Unexpected immediate return from docker_info_source.lookup()", - request.container_id.c_str()); - } -} - void docker::update_with_size(const std::string &container_id) { auto cb = [this](const docker_lookup_request& request, const sinsp_container_info& res) { diff --git a/userspace/libsinsp/container_engine/docker_linux.cpp b/userspace/libsinsp/container_engine/docker_linux.cpp index 2648de922..9e31d7348 100644 --- a/userspace/libsinsp/container_engine/docker_linux.cpp +++ b/userspace/libsinsp/container_engine/docker_linux.cpp @@ -36,11 +36,6 @@ constexpr const cgroup_layout DOCKER_CGROUP_LAYOUT[] = { std::string docker::m_docker_sock = "/var/run/docker.sock"; -void docker::cleanup() -{ - m_docker_info_source.reset(NULL); -} - bool docker::detect_docker(const sinsp_threadinfo *tinfo, std::string &container_id, std::string &container_name) { if(matches_runc_cgroups(tinfo, DOCKER_CGROUP_LAYOUT, container_id)) diff --git a/userspace/libsinsp/container_engine/docker_win.cpp b/userspace/libsinsp/container_engine/docker_win.cpp index 965e792d7..5c255d76b 100644 --- a/userspace/libsinsp/container_engine/docker_win.cpp +++ b/userspace/libsinsp/container_engine/docker_win.cpp @@ -29,11 +29,6 @@ docker::docker(container_cache_interface& cache, const wmi_handle_source& wmi_so { } -void docker::cleanup() -{ - g_docker_info_source.reset(NULL); -} - bool docker::detect_docker(sinsp_threadinfo *tinfo, std::string &container_id, std::string &container_name) { wh_docker_container_info wcinfo = wh_docker_resolve_pid(m_wmi_handle_source.get_wmi_handle(), tinfo->m_pid); From 931f2987e9964faf728dc15d0393eeb14e494941 Mon Sep 17 00:00:00 2001 From: Grzegorz Nosek Date: Fri, 16 Jul 2021 18:49:45 +0200 Subject: [PATCH 079/148] Split Linux/Windows docker implementations They have different interfaces and different implementations, keeping both under the same name doesn't really avoid any duplication Signed-off-by: Grzegorz Nosek --- userspace/libsinsp/CMakeLists.txt | 7 +- userspace/libsinsp/container.cpp | 14 ++- userspace/libsinsp/container_engine/docker.h | 91 ------------------- .../docker_linux.cpp} | 47 +++++----- .../container_engine/docker/docker_linux.h | 28 ++++++ .../{ => docker}/docker_win.cpp | 17 +++- .../container_engine/docker/docker_win.h | 24 +++++ .../container_engine/docker_linux.cpp | 50 ---------- userspace/libsinsp/parsers.cpp | 2 +- 9 files changed, 101 insertions(+), 179 deletions(-) delete mode 100644 userspace/libsinsp/container_engine/docker.h rename userspace/libsinsp/container_engine/{docker_common.cpp => docker/docker_linux.cpp} (51%) create mode 100644 userspace/libsinsp/container_engine/docker/docker_linux.h rename userspace/libsinsp/container_engine/{ => docker}/docker_win.cpp (69%) create mode 100644 userspace/libsinsp/container_engine/docker/docker_win.h delete mode 100644 userspace/libsinsp/container_engine/docker_linux.cpp diff --git a/userspace/libsinsp/CMakeLists.txt b/userspace/libsinsp/CMakeLists.txt index 0007e4e6a..f14c54c63 100644 --- a/userspace/libsinsp/CMakeLists.txt +++ b/userspace/libsinsp/CMakeLists.txt @@ -139,16 +139,15 @@ if(NOT MINIMAL_BUILD) mesos_http.cpp mesos_state.cpp sinsp_curl.cpp - container_engine/docker_common.cpp container_engine/docker/async_source.cpp container_engine/docker/base.cpp) if(WIN32) list(APPEND SINSP_SOURCES - container_engine/docker_win.cpp - container_engine/docker/connection_win.cpp) + container_engine/docker/connection_win.cpp + container_engine/docker/docker_win.cpp) else() list(APPEND SINSP_SOURCES - container_engine/docker_linux.cpp + container_engine/docker/docker_linux.cpp container_engine/docker/connection_linux.cpp container_engine/libvirt_lxc.cpp container_engine/lxc.cpp diff --git a/userspace/libsinsp/container.cpp b/userspace/libsinsp/container.cpp index 6481d4876..d9deea596 100644 --- a/userspace/libsinsp/container.cpp +++ b/userspace/libsinsp/container.cpp @@ -21,7 +21,11 @@ limitations under the License. #ifdef HAS_CAPTURE #include "container_engine/cri.h" #endif // HAS_CAPTURE -#include "container_engine/docker.h" +#ifdef _WIN32 +#include "container_engine/docker/docker_win.h" +#else +#include "container_engine/docker/docker_linux.h" +#endif #include "container_engine/rkt.h" #include "container_engine/libvirt_lxc.h" #include "container_engine/lxc.h" @@ -524,14 +528,14 @@ void sinsp_container_manager::create_engines() #ifndef MINIMAL_BUILD #ifdef CYGWING_AGENT { - auto docker_engine = std::make_shared(*this, m_inspector /*wmi source*/); + auto docker_engine = std::make_shared(*this, m_inspector /*wmi source*/); m_container_engines.push_back(docker_engine); m_container_engine_by_type[CT_DOCKER] = docker_engine; } #else #ifndef _WIN32 { - auto docker_engine = std::make_shared(*this); + auto docker_engine = std::make_shared(*this); m_container_engines.push_back(docker_engine); m_container_engine_by_type[CT_DOCKER] = docker_engine; } @@ -604,8 +608,8 @@ void sinsp_container_manager::cleanup() void sinsp_container_manager::set_docker_socket_path(std::string socket_path) { -#if !defined(MINIMAL_BUILD) && defined(HAS_CAPTURE) - libsinsp::container_engine::docker::set_docker_sock(std::move(socket_path)); +#if !defined(MINIMAL_BUILD) && defined(HAS_CAPTURE) && !defined(_WIN32) + libsinsp::container_engine::docker_linux::set_docker_sock(std::move(socket_path)); #endif } diff --git a/userspace/libsinsp/container_engine/docker.h b/userspace/libsinsp/container_engine/docker.h deleted file mode 100644 index 734fc98e5..000000000 --- a/userspace/libsinsp/container_engine/docker.h +++ /dev/null @@ -1,91 +0,0 @@ -/* -Copyright (C) 2021 The Falco Authors. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. - -*/ - -#pragma once -#ifndef MINIMAL_BUILD -#ifndef _WIN32 - -#include -#include -#include -#include - -#if !defined(_WIN32) -#include -#include -#include -#endif - -#include "json/json.h" - -#include "async_key_value_source.h" - -#include "container.h" -#include "container_info.h" - -#include "container_engine/docker/async_source.h" -#include "container_engine/docker/base.h" -#include "container_engine/docker/connection.h" -#include "container_engine/docker/lookup_request.h" -#include "container_engine/container_engine_base.h" -#include "container_engine/sinsp_container_type.h" -#include "container_engine/wmi_handle_source.h" - -class sinsp; -class sinsp_threadinfo; - -namespace libsinsp { -namespace container_engine { - -class docker : public docker_base -{ -public: - -#ifdef _WIN32 - docker(container_cache_interface &cache, const wmi_handle_source&); -#else - docker(container_cache_interface &cache) : docker_base(cache) - {} -#endif - - // Container name only set for windows. For linux name must be fetched via lookup - static bool detect_docker(const sinsp_threadinfo* tinfo, std::string& container_id, std::string &container_name); - -#ifndef _WIN32 - static void set_docker_sock(std::string docker_sock) { - m_docker_sock = std::move(docker_sock); - } -#endif - -protected: - static std::string s_incomplete_info_name; -#ifdef _WIN32 - const wmi_handle_source& m_wmi_handle_source; -#else - static std::string m_docker_sock; -#endif - -private: - // implement container_engine_base - bool resolve(sinsp_threadinfo *tinfo, bool query_os_for_missing_info) override; - void update_with_size(const std::string& container_id) override; -}; -} -} - -#endif // _WIN32 -#endif // MINIMAL_BUILD \ No newline at end of file diff --git a/userspace/libsinsp/container_engine/docker_common.cpp b/userspace/libsinsp/container_engine/docker/docker_linux.cpp similarity index 51% rename from userspace/libsinsp/container_engine/docker_common.cpp rename to userspace/libsinsp/container_engine/docker/docker_linux.cpp index b418dd65d..8ae752152 100644 --- a/userspace/libsinsp/container_engine/docker_common.cpp +++ b/userspace/libsinsp/container_engine/docker/docker_linux.cpp @@ -14,43 +14,46 @@ See the License for the specific language governing permissions and limitations under the License. */ +#include "container_engine/docker/docker_linux.h" -#ifndef _WIN32 - -#include "container_engine/docker.h" -#include "cgroup_list_counter.h" -#include "sinsp.h" +#include "runc.h" #include "sinsp_int.h" -#include "container.h" -#include "utils.h" -#include using namespace libsinsp::container_engine; +using namespace libsinsp::runc; + +namespace { +constexpr const cgroup_layout DOCKER_CGROUP_LAYOUT[] = { + {"/", ""}, // non-systemd docker + {"/docker-", ".scope"}, // systemd docker + {nullptr, nullptr} +}; +} -std::string docker::s_incomplete_info_name = "incomplete"; +std::string docker_linux::m_docker_sock = "/var/run/docker.sock"; -bool docker::resolve(sinsp_threadinfo *tinfo, bool query_os_for_missing_info) +bool docker_linux::resolve(sinsp_threadinfo *tinfo, bool query_os_for_missing_info) { - std::string container_id, container_name; + std::string container_id; - if(!detect_docker(tinfo, container_id, container_name)) + if(!matches_runc_cgroups(tinfo, DOCKER_CGROUP_LAYOUT, container_id)) { return false; } - docker_lookup_request request(container_id, m_docker_sock, false /*don't request size*/); - - return resolve_impl(tinfo, request, query_os_for_missing_info); - + return resolve_impl(tinfo, docker_lookup_request( + container_id, + m_docker_sock, + false), query_os_for_missing_info); } -void docker::update_with_size(const std::string &container_id) +void docker_linux::update_with_size(const std::string &container_id) { - auto cb = [this](const docker_lookup_request& request, const sinsp_container_info& res) { + auto cb = [this](const docker_lookup_request& instruction, const sinsp_container_info& res) { g_logger.format(sinsp_logger::SEV_DEBUG, "docker_async (%s): with size callback result=%d", - request.container_id.c_str(), + instruction.container_id.c_str(), res.m_lookup_state); sinsp_container_info::ptr_t updated = make_shared(res); @@ -62,8 +65,6 @@ void docker::update_with_size(const std::string &container_id) container_id.c_str()); sinsp_container_info result; - docker_lookup_request request(container_id, m_docker_sock, true /*request rw size*/); - (void)m_docker_info_source->lookup(request, result, cb); + docker_lookup_request instruction(container_id, m_docker_sock, true /*request rw size*/); + (void)m_docker_info_source->lookup(instruction, result, cb); } - -#endif // _WIN32 diff --git a/userspace/libsinsp/container_engine/docker/docker_linux.h b/userspace/libsinsp/container_engine/docker/docker_linux.h new file mode 100644 index 000000000..0cd53614e --- /dev/null +++ b/userspace/libsinsp/container_engine/docker/docker_linux.h @@ -0,0 +1,28 @@ +#pragma once + +#include "container_engine/container_engine_base.h" +#include "container_engine/docker/base.h" + +namespace libsinsp { +namespace container_engine { + +class docker_linux : public docker_base { +public: + docker_linux(container_cache_interface& cache) : docker_base(cache) {} + + static void set_docker_sock(std::string docker_sock) + { + m_docker_sock = std::move(docker_sock); + } + + // implement container_engine_base + bool resolve(sinsp_threadinfo *tinfo, bool query_os_for_missing_info) override; + + void update_with_size(const std::string& container_id) override; + +private: + static std::string m_docker_sock; +}; + +} +} diff --git a/userspace/libsinsp/container_engine/docker_win.cpp b/userspace/libsinsp/container_engine/docker/docker_win.cpp similarity index 69% rename from userspace/libsinsp/container_engine/docker_win.cpp rename to userspace/libsinsp/container_engine/docker/docker_win.cpp index 5c255d76b..7d7908166 100644 --- a/userspace/libsinsp/container_engine/docker_win.cpp +++ b/userspace/libsinsp/container_engine/docker/docker_win.cpp @@ -23,13 +23,13 @@ limitations under the License. using namespace libsinsp::container_engine; -docker::docker(container_cache_interface& cache, const wmi_handle_source& wmi_source) : +docker_win::docker_win(container_cache_interface& cache, const wmi_handle_source& wmi_source) : container_engine_base(cache), m_wmi_handle_source(wmi_source) { } -bool docker::detect_docker(sinsp_threadinfo *tinfo, std::string &container_id, std::string &container_name) +bool docker_win::resolve(sinsp_threadinfo *tinfo, bool query_os_for_missing_info) { wh_docker_container_info wcinfo = wh_docker_resolve_pid(m_wmi_handle_source.get_wmi_handle(), tinfo->m_pid); if(!wcinfo.m_res) @@ -37,10 +37,17 @@ bool docker::detect_docker(sinsp_threadinfo *tinfo, std::string &container_id, s return false; } - container_id = wcinfo.m_container_id; - container_name = wcinfo.m_container_name; + std::string container_id = wcinfo.m_container_id; - return true; + return resolve_impl(tinfo, docker_async_instruction( + container_id, + "", + false), query_os_for_missing_info); +} + +void docker_win::update_with_size(const std::string &container_id) +{ + // not supported } #endif // CYGWING_AGENT diff --git a/userspace/libsinsp/container_engine/docker/docker_win.h b/userspace/libsinsp/container_engine/docker/docker_win.h new file mode 100644 index 000000000..7d51e8243 --- /dev/null +++ b/userspace/libsinsp/container_engine/docker/docker_win.h @@ -0,0 +1,24 @@ +#pragma once + +#include "container_engine/container_engine_base.h" +#include "container_engine/docker_base.h" + +namespace libsinsp { +namespace container_engine { + +class docker_win : public docker_base +{ +public: + docker_win(container_cache_interface &cache, const wmi_handle_source&); + + // implement container_engine_base + bool resolve(sinsp_threadinfo *tinfo, bool query_os_for_missing_info) override; + void update_with_size(const std::string& container_id) override; + +private: + static std::string s_incomplete_info_name; + const wmi_handle_source& m_wmi_handle_source; +}; + +} +} diff --git a/userspace/libsinsp/container_engine/docker_linux.cpp b/userspace/libsinsp/container_engine/docker_linux.cpp deleted file mode 100644 index 9e31d7348..000000000 --- a/userspace/libsinsp/container_engine/docker_linux.cpp +++ /dev/null @@ -1,50 +0,0 @@ -/* -Copyright (C) 2021 The Falco Authors. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. - -*/ - -#include "container_engine/docker.h" - -#include "runc.h" -#include "container_engine/mesos.h" -#include "sinsp.h" -#include "sinsp_int.h" - -using namespace libsinsp::container_engine; -using namespace libsinsp::runc; - -namespace { - -constexpr const cgroup_layout DOCKER_CGROUP_LAYOUT[] = { - {"/", ""}, // non-systemd docker - {"/docker-", ".scope"}, // systemd docker - {nullptr, nullptr} -}; -} - -std::string docker::m_docker_sock = "/var/run/docker.sock"; - -bool docker::detect_docker(const sinsp_threadinfo *tinfo, std::string &container_id, std::string &container_name) -{ - if(matches_runc_cgroups(tinfo, DOCKER_CGROUP_LAYOUT, container_id)) - { - // The container name is only available in windows - container_name = s_incomplete_info_name; - - return true; - } - - return false; -} diff --git a/userspace/libsinsp/parsers.cpp b/userspace/libsinsp/parsers.cpp index 566335bac..154d6f4b6 100644 --- a/userspace/libsinsp/parsers.cpp +++ b/userspace/libsinsp/parsers.cpp @@ -43,7 +43,7 @@ bool should_drop(sinsp_evt *evt); #endif #include "sinsp_int.h" -#include "container_engine/docker.h" +#include "container_engine/docker/async_source.h" extern sinsp_protodecoder_list g_decoderlist; extern sinsp_evttables g_infotables; From 83f460cc5ba39cd09f924b4d05a1ffd13eec07de Mon Sep 17 00:00:00 2001 From: Grzegorz Nosek Date: Mon, 19 Jul 2021 15:06:38 +0200 Subject: [PATCH 080/148] Split docker_async_source::parse_docker into helper methods Signed-off-by: Grzegorz Nosek --- .../container_engine/docker/async_source.cpp | 281 ++++++++++-------- .../container_engine/docker/async_source.h | 12 + userspace/libsinsp/utils.cpp | 16 + userspace/libsinsp/utils.h | 5 + 4 files changed, 188 insertions(+), 126 deletions(-) diff --git a/userspace/libsinsp/container_engine/docker/async_source.cpp b/userspace/libsinsp/container_engine/docker/async_source.cpp index fa36c7828..afb913668 100644 --- a/userspace/libsinsp/container_engine/docker/async_source.cpp +++ b/userspace/libsinsp/container_engine/docker/async_source.cpp @@ -375,6 +375,160 @@ void docker_async_source::set_query_image_info(bool query_image_info) m_query_image_info = query_image_info; } +void docker_async_source::fetch_image_info(const docker_lookup_request& request, sinsp_container_info& container) +{ + Json::Reader reader; + + g_logger.format(sinsp_logger::SEV_DEBUG, + "docker_async (%s) image (%s): Fetching image info", + request.container_id.c_str(), + container.m_imageid.c_str()); + + std::string img_json; + std::string url = "/images/" + container.m_imageid + "/json?digests=1"; + + g_logger.format(sinsp_logger::SEV_DEBUG, + "docker_async url: %s", + url.c_str()); + + if(!(m_connection.get_docker(request, url, img_json) == docker_connection::RESP_OK)) + { + g_logger.format(sinsp_logger::SEV_ERROR, + "docker_async (%s) image (%s): Could not fetch image info", + request.container_id.c_str(), + container.m_imageid.c_str()); + return; + } + + g_logger.format(sinsp_logger::SEV_DEBUG, + "docker_async (%s) image (%s): Image info fetch returned \"%s\"", + request.container_id.c_str(), + container.m_imageid.c_str(), + img_json.c_str()); + + Json::Value img_root; + if(!reader.parse(img_json, img_root)) + { + g_logger.format(sinsp_logger::SEV_ERROR, + "docker_async (%s) image (%s): Could not parse json image info \"%s\"", + request.container_id.c_str(), + container.m_imageid.c_str(), + img_json.c_str()); + return; + } + + parse_image_info(container, img_root); +} + +void docker_async_source::parse_image_info(sinsp_container_info& container, const Json::Value& img) +{ + // img_root["RepoDigests"] contains only digests for images pulled from registries. + // If an image gets retagged and is never pushed to any registry, we will not find + // that entry in container.m_imagerepo. Also, for locally built images we have the + // same issue. This leads to container.m_imagedigest being empty as well. + // + // Each individual digest looks like e.g. + // "docker.io/library/redis@sha256:b6a9fc3535388a6fc04f3bdb83fb4d9d0b4ffd85e7609a6ff2f0f731427823e3" + // so we need to split it at the `@` (the part before is the repo, + // the part after is the digest) + std::unordered_set imageDigestSet; + for(const auto& rdig : img["RepoDigests"]) + { + if(rdig.isString()) + { + std::string repodigest = rdig.asString(); + std::string digest = repodigest.substr(repodigest.find('@')+1); + imageDigestSet.insert(digest); + if(container.m_imagerepo.empty()) + { + container.m_imagerepo = repodigest.substr(0, repodigest.find('@')); + } + if(repodigest.find(container.m_imagerepo) != std::string::npos) + { + container.m_imagedigest = digest; + break; + } + } + } + + // fix image digest for locally tagged images or multiple repo digests. + // Case 1: One repo digest with many tags. + // Case 2: Many repo digests with the same digest value. + if(container.m_imagedigest.empty() && imageDigestSet.size() == 1) { + container.m_imagedigest = *imageDigestSet.begin(); + } + + for(const auto& rtag : img["RepoTags"]) + { + if(rtag.isString()) + { + std::string repotag = rtag.asString(); + if(container.m_imagerepo.empty()) + { + container.m_imagerepo = repotag.substr(0, repotag.rfind(':')); + } + if(repotag.find(container.m_imagerepo) != std::string::npos) + { + container.m_imagetag = repotag.substr(repotag.rfind(':')+1); + break; + } + } + } +} + +void docker_async_source::get_image_info(const docker_lookup_request& request, sinsp_container_info& container, const Json::Value& root) +{ + container.m_image = root["Config"]["Image"].asString(); + + std::string imgstr = root["Image"].asString(); + size_t cpos = imgstr.find(':'); + if(cpos != std::string::npos) + { + container.m_imageid = imgstr.substr(cpos + 1); + } + + // containers can be spawned using just the imageID as image name, + // with or without the hash prefix (e.g. sha256:) + // + // e.g. an image with the id `sha256:ddcca4b8a6f0367b5de2764dfe76b0a4bfa6d75237932185923705da47004347` + // can be used to run a container as: + // - docker run sha256:ddcca4b8a6f0367b5de2764dfe76b0a4bfa6d75237932185923705da47004347 + // - docker run ddcca4b8a6f0367b5de2764dfe76b0a4bfa6d75237932185923705da47004347 + // - docker run sha256:ddcca4 + // - docker run ddcca4 + // + // in all these cases we need to determine the image repo/tag + // via the API (in `fetch_image_info()`) + // + // otherwise we assume the name passed to `docker run` + // (available in container.m_image) is the repo name like `redis` + // and use that to determine the name and tag + bool no_name = sinsp_utils::startswith(container.m_imageid, container.m_image) || + sinsp_utils::startswith(imgstr, container.m_image); + + if(!no_name || !m_query_image_info) + { + std::string hostname, port; + sinsp_utils::split_container_image(container.m_image, + hostname, + port, + container.m_imagerepo, + container.m_imagetag, + container.m_imagedigest, + false); + } + + if(m_query_image_info && !container.m_imageid.empty() && + (no_name || container.m_imagedigest.empty() || container.m_imagetag.empty())) + { + fetch_image_info(request, container); + } + + if(container.m_imagetag.empty()) + { + container.m_imagetag = "latest"; + } +} void docker_async_source::parse_json_mounts(const Json::Value &mnt_obj, vector &mounts) { if(!mnt_obj.isNull() && mnt_obj.isArray()) @@ -450,136 +604,11 @@ bool docker_async_source::parse_docker(const docker_lookup_request& request, sin return false; } - const Json::Value& config_obj = root["Config"]; - - container.m_image = config_obj["Image"].asString(); - - string imgstr = root["Image"].asString(); - size_t cpos = imgstr.find(":"); - if(cpos != string::npos) - { - container.m_imageid = imgstr.substr(cpos + 1); - } + get_image_info(request, container, root); const Json::Value& config_obj = root["Config"]; parse_health_probes(config_obj, container); - // containers can be spawned using just the imageID as image name, - // with or without the hash prefix (e.g. sha256:) - bool no_name = !container.m_imageid.empty() && - strncmp(container.m_image.c_str(), container.m_imageid.c_str(), - MIN(container.m_image.length(), container.m_imageid.length())) == 0; - no_name |= !imgstr.empty() && - strncmp(container.m_image.c_str(), imgstr.c_str(), - MIN(container.m_image.length(), imgstr.length())) == 0; - - if(!no_name || !m_query_image_info) - { - string hostname, port; - sinsp_utils::split_container_image(container.m_image, - hostname, - port, - container.m_imagerepo, - container.m_imagetag, - container.m_imagedigest, - false); - } - - if(m_query_image_info && !container.m_imageid.empty() && - (no_name || container.m_imagedigest.empty() || (!container.m_imagedigest.empty() && container.m_imagetag.empty()))) - { - g_logger.format(sinsp_logger::SEV_DEBUG, - "docker_async (%s) image (%s): Fetching image info", - request.container_id.c_str(), - container.m_imageid.c_str()); - - string img_json; - std::string url = "/images/" + container.m_imageid + "/json?digests=1"; - - g_logger.format(sinsp_logger::SEV_DEBUG, - "docker_async url: %s", - url.c_str()); - - if(m_connection.get_docker(request, url, img_json) == docker_connection::docker_response::RESP_OK) - { - g_logger.format(sinsp_logger::SEV_DEBUG, - "docker_async (%s) image (%s): Image info fetch returned \"%s\"", - request.container_id.c_str(), - container.m_imageid.c_str(), - img_json.c_str()); - - Json::Value img_root; - if(reader.parse(img_json, img_root)) - { - // img_root["RepoDigests"] contains only digests for images pulled from registries. - // If an image gets retagged and is never pushed to any registry, we will not find - // that entry in container.m_imagerepo. Also, for locally built images we have the - // same issue. This leads to container.m_imagedigest being empty as well. - unordered_set imageDigestSet; - for(const auto& rdig : img_root["RepoDigests"]) - { - if(rdig.isString()) - { - string repodigest = rdig.asString(); - string digest = repodigest.substr(repodigest.find('@')+1); - imageDigestSet.insert(digest); - if(container.m_imagerepo.empty()) - { - container.m_imagerepo = repodigest.substr(0, repodigest.find('@')); - } - if(repodigest.find(container.m_imagerepo) != string::npos) - { - container.m_imagedigest = digest; - break; - } - } - } - for(const auto& rtag : img_root["RepoTags"]) - { - if(rtag.isString()) - { - string repotag = rtag.asString(); - if(container.m_imagerepo.empty()) - { - container.m_imagerepo = repotag.substr(0, repotag.rfind(":")); - } - if(repotag.find(container.m_imagerepo) != string::npos) - { - container.m_imagetag = repotag.substr(repotag.rfind(":")+1); - break; - } - } - } - // fix image digest for locally tagged images or multiple repo digests. - // Case 1: One repo digest with many tags. - // Case 2: Many repo digests with the same digest value. - if(container.m_imagedigest.empty() && imageDigestSet.size() == 1) { - container.m_imagedigest = *imageDigestSet.begin(); - } - } - else - { - g_logger.format(sinsp_logger::SEV_ERROR, - "docker_async (%s) image (%s): Could not parse json image info \"%s\"", - request.container_id.c_str(), - container.m_imageid.c_str(), - img_json.c_str()); - } - } - else - { - g_logger.format(sinsp_logger::SEV_ERROR, - "docker_async (%s) image (%s): Could not fetch image info", - request.container_id.c_str(), - container.m_imageid.c_str()); - } - - } - if(container.m_imagetag.empty()) - { - container.m_imagetag = "latest"; - } - container.m_full_id = root["Id"].asString(); container.m_name = root["Name"].asString(); // k8s Docker container names could have '/' as the first character. diff --git a/userspace/libsinsp/container_engine/docker/async_source.h b/userspace/libsinsp/container_engine/docker/async_source.h index f9efbc8f5..926115148 100644 --- a/userspace/libsinsp/container_engine/docker/async_source.h +++ b/userspace/libsinsp/container_engine/docker/async_source.h @@ -62,6 +62,18 @@ class docker_async_source : public sysdig::async_key_value_source/json, + // or one of the items from the result of /images/json), find + // the image digest, repo and repo tag + static void parse_image_info(sinsp_container_info& container, const Json::Value& img); + + // Fetch the image info for the current container's m_imageid + void fetch_image_info(const docker_lookup_request& request, sinsp_container_info& container); + container_cache_interface *m_cache; docker_connection m_connection; static bool m_query_image_info; diff --git a/userspace/libsinsp/utils.cpp b/userspace/libsinsp/utils.cpp index 50ce73e66..2de13f14e 100644 --- a/userspace/libsinsp/utils.cpp +++ b/userspace/libsinsp/utils.cpp @@ -1517,6 +1517,22 @@ bool sinsp_utils::endswith(const char *str, const char *ending, uint32_t lstr, u return 0; } +bool sinsp_utils::startswith(const std::string& s, const std::string& prefix) +{ + if(prefix.empty()) + { + return false; + } + + size_t prefix_len = prefix.length(); + if(s.length() < prefix_len) + { + return false; + } + + return strncmp(s.c_str(), prefix.c_str(), prefix_len) == 0; +} + /////////////////////////////////////////////////////////////////////////////// // sinsp_numparser implementation diff --git a/userspace/libsinsp/utils.h b/userspace/libsinsp/utils.h index 28a62c20a..7fd94f33c 100644 --- a/userspace/libsinsp/utils.h +++ b/userspace/libsinsp/utils.h @@ -76,6 +76,11 @@ class sinsp_utils static bool endswith(const std::string& str, const std::string& ending); static bool endswith(const char *str, const char *ending, uint32_t lstr, uint32_t lend); + // + // Check if string starts with another + // + static bool startswith(const std::string& s, const std::string& prefix); + // // Concatenate two paths and puts the result in "target". // If path2 is relative, the concatenation happens and the result is true. From 9d2082aa5a84dbd42890c4b5aae73c5ec38e2b65 Mon Sep 17 00:00:00 2001 From: Luca Guerra Date: Tue, 9 Nov 2021 08:36:17 +0000 Subject: [PATCH 081/148] chore(libscap): remove unused variable sh from scap_read_init Signed-off-by: Luca Guerra --- userspace/libscap/scap_savefile.c | 1 - 1 file changed, 1 deletion(-) diff --git a/userspace/libscap/scap_savefile.c b/userspace/libscap/scap_savefile.c index aad9737c1..3d09833b8 100755 --- a/userspace/libscap/scap_savefile.c +++ b/userspace/libscap/scap_savefile.c @@ -2563,7 +2563,6 @@ int32_t scap_read_section_header(scap_t *handle, gzFile f) int32_t scap_read_init(scap_t *handle, gzFile f) { block_header bh; - section_header_block sh; uint32_t bt; size_t readsize; size_t toread; From fcffb427b75569706e83b11c62cc7673de38aab9 Mon Sep 17 00:00:00 2001 From: Federico Di Pierro Date: Tue, 9 Nov 2021 10:23:08 +0100 Subject: [PATCH 082/148] fix(userspace/libsinsp): properly use dlerror() instead of errno when catching errors of dlopen. Signed-off-by: Federico Di Pierro --- userspace/libsinsp/plugin.cpp | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/userspace/libsinsp/plugin.cpp b/userspace/libsinsp/plugin.cpp index 9d001e69f..fe466f35d 100755 --- a/userspace/libsinsp/plugin.cpp +++ b/userspace/libsinsp/plugin.cpp @@ -411,7 +411,7 @@ std::shared_ptr sinsp_plugin::create_plugin(string &filepath, cons #endif if(handle == NULL) { - errstr = "error loading plugin " + filepath + ": " + strerror(errno); + errstr = "error loading plugin " + filepath + ": " + dlerror(); return ret; } From b257f5f65ad92f70f748f5ec8e7ba338a8319802 Mon Sep 17 00:00:00 2001 From: Federico Di Pierro Date: Tue, 2 Nov 2021 10:38:42 +0100 Subject: [PATCH 083/148] new(userspace/libsinsp): properly support filtering on rawarg.(rel)path. Signed-off-by: Federico Di Pierro --- userspace/libsinsp/filter.cpp | 4 ++-- userspace/libsinsp/value_parser.cpp | 2 ++ 2 files changed, 4 insertions(+), 2 deletions(-) diff --git a/userspace/libsinsp/filter.cpp b/userspace/libsinsp/filter.cpp index 2a01a5171..3117a93b9 100644 --- a/userspace/libsinsp/filter.cpp +++ b/userspace/libsinsp/filter.cpp @@ -408,6 +408,8 @@ bool flt_compare(cmpop op, ppm_param_type type, void* operand1, void* operand2, case PT_ABSTIME: return flt_compare_uint64(op, *(uint64_t*)operand1, *(uint64_t*)operand2); case PT_CHARBUF: + case PT_FSPATH: + case PT_FSRELPATH: return flt_compare_string(op, (char*)operand1, (char*)operand2); case PT_BYTEBUF: return flt_compare_buffer(op, (char*)operand1, (char*)operand2, op1_len, op2_len); @@ -416,9 +418,7 @@ bool flt_compare(cmpop op, ppm_param_type type, void* operand1, void* operand2, case PT_SOCKADDR: case PT_SOCKTUPLE: case PT_FDLIST: - case PT_FSPATH: case PT_SIGSET: - case PT_FSRELPATH: default: ASSERT(false); return false; diff --git a/userspace/libsinsp/value_parser.cpp b/userspace/libsinsp/value_parser.cpp index 119d3e173..30a0ec7d9 100644 --- a/userspace/libsinsp/value_parser.cpp +++ b/userspace/libsinsp/value_parser.cpp @@ -111,6 +111,8 @@ size_t sinsp_filter_value_parser::string_to_rawval(const char* str, uint32_t len case PT_CHARBUF: case PT_SOCKADDR: case PT_SOCKFAMILY: + case PT_FSPATH: + case PT_FSRELPATH: { len = (uint32_t)strlen(str); if(len >= max_len) From c7524e8e13dcb499ada42052a5c0a9f0b02045c5 Mon Sep 17 00:00:00 2001 From: Mark Stemm Date: Thu, 11 Nov 2021 10:09:48 -0800 Subject: [PATCH 084/148] Add Mark Stemm as libs owner/reviewer I know the code fairly well and would be happy to help out with reviewing PRs. Signed-off-by: Mark Stemm --- OWNERS | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/OWNERS b/OWNERS index b6121d987..a145a0a0e 100644 --- a/OWNERS +++ b/OWNERS @@ -4,9 +4,11 @@ approvers: - leogr - gnosek - ldegio + - mstemm reviewers: - fntlnz - leodido - leogr - gnosek - - ldegio \ No newline at end of file + - ldegio + - mstemm From f3f41828d65c93c48c331a5958ac7974b2a9a61a Mon Sep 17 00:00:00 2001 From: Federico Di Pierro Date: Mon, 27 Sep 2021 12:52:03 +0200 Subject: [PATCH 085/148] fix(driver): properly support AF_UNIX abstract socket addresses; see https://man7.org/linux/man-pages/man7/unix.7.html. Signed-off-by: Federico Di Pierro --- driver/bpf/filler_helpers.h | 30 ++++++++++++++++++++++++------ driver/ppm_events.c | 31 +++++++++++++++++++++++++------ 2 files changed, 49 insertions(+), 12 deletions(-) diff --git a/driver/bpf/filler_helpers.h b/driver/bpf/filler_helpers.h index 348e461f3..25cefc20d 100644 --- a/driver/bpf/filler_helpers.h +++ b/driver/bpf/filler_helpers.h @@ -406,6 +406,28 @@ static __always_inline u32 bpf_compute_snaplen(struct filler_data *data, return res; } +static __always_inline int unix_socket_path(char *dest, const char *user_ptr) { + int res = bpf_probe_read_str(dest, + UNIX_PATH_MAX, + user_ptr); + /* + * Extract from: https://man7.org/linux/man-pages/man7/unix.7.html + * an abstract socket address is distinguished (from a + * pathname socket) by the fact that sun_path[0] is a null byte + * ('\0'). The socket's address in this namespace is given by + * the additional bytes in sun_path that are covered by the + * specified length of the address structure. + */ + if (res == 1) { + dest[0] = '@'; + res = bpf_probe_read_str(dest + 1, + UNIX_PATH_MAX, + user_ptr + 1); + res++; // account for '@' + } + return res; +} + static __always_inline u16 bpf_pack_addr(struct filler_data *data, struct sockaddr *usrsockaddr, int ulen) @@ -487,9 +509,7 @@ static __always_inline u16 bpf_pack_addr(struct filler_data *data, data->buf[data->state->tail_ctx.curoff & SCRATCH_SIZE_HALF] = socket_family_to_scap(family); - res = bpf_probe_read_str(&data->buf[(data->state->tail_ctx.curoff + 1) & SCRATCH_SIZE_HALF], - UNIX_PATH_MAX, - usrsockaddr_un->sun_path); + res = unix_socket_path(&data->buf[(data->state->tail_ctx.curoff + 1) & SCRATCH_SIZE_HALF], usrsockaddr_un->sun_path); size += res; @@ -697,9 +717,7 @@ static __always_inline long bpf_fd_to_socktuple(struct filler_data *data, us_name = usrsockaddr_un->sun_path; } - int res = bpf_probe_read_str(&data->buf[(data->state->tail_ctx.curoff + 1 + 8 + 8) & SCRATCH_SIZE_HALF], - UNIX_PATH_MAX, - us_name); + int res = unix_socket_path(&data->buf[(data->state->tail_ctx.curoff + 1 + 8 + 8) & SCRATCH_SIZE_HALF], us_name); size += res; diff --git a/driver/ppm_events.c b/driver/ppm_events.c index daf2dc426..eecc20010 100644 --- a/driver/ppm_events.c +++ b/driver/ppm_events.c @@ -885,6 +885,28 @@ static struct socket *ppm_sockfd_lookup_light(int fd, int *err, int *fput_needed } */ +static void unix_socket_path(char *dest, const char *path) +{ + if (path[0] == '\0') { + /* + * Extract from: https://man7.org/linux/man-pages/man7/unix.7.html + * an abstract socket address is distinguished (from a + * pathname socket) by the fact that sun_path[0] is a null byte + * ('\0'). The socket's address in this namespace is given by + * the additional bytes in sun_path that are covered by the + * specified length of the address structure. + */ + snprintf(dest, + UNIX_PATH_MAX, + "@%s", + path + 1); + } else { + dest = strncpy(dest, + path, + UNIX_PATH_MAX); /* we assume this will be smaller than (targetbufsize - (1 + 8 + 8)) */ + } +} + /* * Convert a sockaddr into our address representation and copy it to * targetbuf @@ -974,9 +996,7 @@ u16 pack_addr(struct sockaddr *usrsockaddr, size = 1; *targetbuf = socket_family_to_scap((u8)family); - dest = strncpy(targetbuf + 1, - usrsockaddr_un->sun_path, - UNIX_PATH_MAX); /* we assume this will be smaller than (targetbufsize - (1 + 8 + 8)) */ + unix_socket_path(targetbuf + 1, usrsockaddr_un->sun_path); dest[UNIX_PATH_MAX - 1] = 0; size += (u16)strlen(dest) + 1; @@ -1233,9 +1253,8 @@ u16 fd_to_socktuple(int fd, } ASSERT(us_name); - dest = strncpy(targetbuf + 1 + 8 + 8, - (char *)us_name, - UNIX_PATH_MAX); /* we assume this will be smaller than (targetbufsize - (1 + 8 + 8)) */ + dest = targetbuf + 1 + 8 + 8; + unix_socket_path(dest, us_name); dest[UNIX_PATH_MAX - 1] = 0; size += strlen(dest) + 1; From 5e017563ac59c43ff61dd74f6e1e418e5192dfcd Mon Sep 17 00:00:00 2001 From: Federico Di Pierro Date: Mon, 27 Sep 2021 16:13:50 +0200 Subject: [PATCH 086/148] fix(driver): properly update dest pointer upon unix_socket_path() call. Signed-off-by: Federico Di Pierro --- driver/ppm_events.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/driver/ppm_events.c b/driver/ppm_events.c index eecc20010..54f4da4af 100644 --- a/driver/ppm_events.c +++ b/driver/ppm_events.c @@ -996,7 +996,9 @@ u16 pack_addr(struct sockaddr *usrsockaddr, size = 1; *targetbuf = socket_family_to_scap((u8)family); - unix_socket_path(targetbuf + 1, usrsockaddr_un->sun_path); + + dest = targetbuf + 1; + unix_socket_path(dest, usrsockaddr_un->sun_path); dest[UNIX_PATH_MAX - 1] = 0; size += (u16)strlen(dest) + 1; @@ -1253,6 +1255,7 @@ u16 fd_to_socktuple(int fd, } ASSERT(us_name); + dest = targetbuf + 1 + 8 + 8; unix_socket_path(dest, us_name); From cf2527eae79a7511cbe0a9eeb6f654e0d0d21cf3 Mon Sep 17 00:00:00 2001 From: Federico Di Pierro Date: Mon, 27 Sep 2021 17:17:37 +0200 Subject: [PATCH 087/148] fix(driver): improve unix_socket_path() API making it more robust. Signed-off-by: Federico Di Pierro Co-authored-by: lucklypse --- driver/bpf/filler_helpers.h | 14 +++++++++----- driver/ppm_events.c | 13 ++++++------- 2 files changed, 15 insertions(+), 12 deletions(-) diff --git a/driver/bpf/filler_helpers.h b/driver/bpf/filler_helpers.h index 25cefc20d..48a143905 100644 --- a/driver/bpf/filler_helpers.h +++ b/driver/bpf/filler_helpers.h @@ -406,9 +406,9 @@ static __always_inline u32 bpf_compute_snaplen(struct filler_data *data, return res; } -static __always_inline int unix_socket_path(char *dest, const char *user_ptr) { +static __always_inline int unix_socket_path(char *dest, const char *user_ptr, size_t size) { int res = bpf_probe_read_str(dest, - UNIX_PATH_MAX, + size, user_ptr); /* * Extract from: https://man7.org/linux/man-pages/man7/unix.7.html @@ -421,7 +421,7 @@ static __always_inline int unix_socket_path(char *dest, const char *user_ptr) { if (res == 1) { dest[0] = '@'; res = bpf_probe_read_str(dest + 1, - UNIX_PATH_MAX, + size - 1, // account for '@' user_ptr + 1); res++; // account for '@' } @@ -509,7 +509,9 @@ static __always_inline u16 bpf_pack_addr(struct filler_data *data, data->buf[data->state->tail_ctx.curoff & SCRATCH_SIZE_HALF] = socket_family_to_scap(family); - res = unix_socket_path(&data->buf[(data->state->tail_ctx.curoff + 1) & SCRATCH_SIZE_HALF], usrsockaddr_un->sun_path); + res = unix_socket_path(&data->buf[(data->state->tail_ctx.curoff + 1) & SCRATCH_SIZE_HALF], + usrsockaddr_un->sun_path, + UNIX_PATH_MAX); size += res; @@ -717,7 +719,9 @@ static __always_inline long bpf_fd_to_socktuple(struct filler_data *data, us_name = usrsockaddr_un->sun_path; } - int res = unix_socket_path(&data->buf[(data->state->tail_ctx.curoff + 1 + 8 + 8) & SCRATCH_SIZE_HALF], us_name); + int res = unix_socket_path(&data->buf[(data->state->tail_ctx.curoff + 1 + 8 + 8) & SCRATCH_SIZE_HALF], + us_name, + UNIX_PATH_MAX); size += res; diff --git a/driver/ppm_events.c b/driver/ppm_events.c index 54f4da4af..26add41c0 100644 --- a/driver/ppm_events.c +++ b/driver/ppm_events.c @@ -885,7 +885,7 @@ static struct socket *ppm_sockfd_lookup_light(int fd, int *err, int *fput_needed } */ -static void unix_socket_path(char *dest, const char *path) +static void unix_socket_path(char *dest, const char *path, size_t size) { if (path[0] == '\0') { /* @@ -897,13 +897,14 @@ static void unix_socket_path(char *dest, const char *path) * specified length of the address structure. */ snprintf(dest, - UNIX_PATH_MAX, + size, "@%s", path + 1); } else { dest = strncpy(dest, path, - UNIX_PATH_MAX); /* we assume this will be smaller than (targetbufsize - (1 + 8 + 8)) */ + size - 1); /* we assume this will be smaller than (targetbufsize - (1 + 8 + 8)) */ + dest[size - 1] = 0; } } @@ -998,9 +999,8 @@ u16 pack_addr(struct sockaddr *usrsockaddr, *targetbuf = socket_family_to_scap((u8)family); dest = targetbuf + 1; - unix_socket_path(dest, usrsockaddr_un->sun_path); + unix_socket_path(dest, usrsockaddr_un->sun_path, UNIX_PATH_MAX); - dest[UNIX_PATH_MAX - 1] = 0; size += (u16)strlen(dest) + 1; break; @@ -1257,9 +1257,8 @@ u16 fd_to_socktuple(int fd, ASSERT(us_name); dest = targetbuf + 1 + 8 + 8; - unix_socket_path(dest, us_name); + unix_socket_path(dest, us_name, UNIX_PATH_MAX); - dest[UNIX_PATH_MAX - 1] = 0; size += strlen(dest) + 1; #endif /* UDIG */ break; From 6e805036b8749339d3ccb7fe8ec25117b5aff7e0 Mon Sep 17 00:00:00 2001 From: Federico Di Pierro Date: Mon, 27 Sep 2021 17:35:53 +0200 Subject: [PATCH 088/148] fix(driver): use snprintf instead of strncpy. Signed-off-by: Federico Di Pierro --- driver/ppm_events.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/driver/ppm_events.c b/driver/ppm_events.c index 26add41c0..c693c96ad 100644 --- a/driver/ppm_events.c +++ b/driver/ppm_events.c @@ -901,10 +901,10 @@ static void unix_socket_path(char *dest, const char *path, size_t size) "@%s", path + 1); } else { - dest = strncpy(dest, - path, - size - 1); /* we assume this will be smaller than (targetbufsize - (1 + 8 + 8)) */ - dest[size - 1] = 0; + snprintf(dest, + size, + "%s", + path); /* we assume this will be smaller than (targetbufsize - (1 + 8 + 8)) */ } } From 340be865d951d1a9ee140daad7fb5a3d40d4d7dc Mon Sep 17 00:00:00 2001 From: Leonardo Grasso Date: Fri, 12 Nov 2021 16:19:57 +0100 Subject: [PATCH 089/148] fix(libsinsp): correct `MINIMAL_BUILD` gate for docker `container_engine/docker/async_source.h` includes curl, which is not included in the minimal build. Signed-off-by: Leonardo Grasso --- userspace/libsinsp/parsers.cpp | 2 ++ 1 file changed, 2 insertions(+) diff --git a/userspace/libsinsp/parsers.cpp b/userspace/libsinsp/parsers.cpp index 154d6f4b6..3202b0672 100644 --- a/userspace/libsinsp/parsers.cpp +++ b/userspace/libsinsp/parsers.cpp @@ -43,7 +43,9 @@ bool should_drop(sinsp_evt *evt); #endif #include "sinsp_int.h" +#if !defined(MINIMAL_BUILD) #include "container_engine/docker/async_source.h" +#endif extern sinsp_protodecoder_list g_decoderlist; extern sinsp_evttables g_infotables; From 580adee54580a33fdf2399ed0d358b9b00e64fca Mon Sep 17 00:00:00 2001 From: Federico Di Pierro Date: Thu, 11 Nov 2021 14:59:55 +0100 Subject: [PATCH 090/148] new(driver): correctly use WEXITSTATUS set of macros to retrieve procexit return code. Moreover, enrich procexit event with more data: * actual return code * if signal-killed, signal that killed the proc, otherwise 0 * whether a coredump was created Added a systype_compat header to define sys/types.h macros, as in kernel space they're not available. The change is backward compatible as we support adding new params to existing events. Signed-off-by: Federico Di Pierro --- driver/CMakeLists.txt | 1 + driver/bpf/fillers.h | 20 ++++++++++++++++++-- driver/event_table.c | 2 +- driver/ppm_fillers.c | 26 +++++++++++++++++++++++++- driver/systype_compat.h | 14 ++++++++++++++ 5 files changed, 59 insertions(+), 4 deletions(-) create mode 100644 driver/systype_compat.h diff --git a/driver/CMakeLists.txt b/driver/CMakeLists.txt index 2cc8fa1b0..14f086fcf 100644 --- a/driver/CMakeLists.txt +++ b/driver/CMakeLists.txt @@ -60,6 +60,7 @@ set(DRIVER_SOURCES ppm_cputime.c ppm_compat_unistd_32.h ppm_version.h + systype_compat.h ) foreach(FILENAME IN LISTS DRIVER_SOURCES) diff --git a/driver/bpf/fillers.h b/driver/bpf/fillers.h index a4377651b..0b2a8f227 100644 --- a/driver/bpf/fillers.h +++ b/driver/bpf/fillers.h @@ -18,7 +18,7 @@ or GPL2.txt for full copies of the license. * probe to build. */ //#define COS_73_WORKAROUND - +#include "../systype_compat.h" #include "../ppm_flag_helpers.h" #include "../ppm_version.h" @@ -3498,7 +3498,23 @@ FILLER(sys_procexit_e, false) exit_code = _READ(task->exit_code); - res = bpf_val_to_ring(data, exit_code); + /* Real exit code */ + res = bpf_val_to_ring(data, __WEXITSTATUS(exit_code)); + if (res != PPM_SUCCESS) + return res; + + /* If signaled -> signum, else 0 */ + if (__WIFSIGNALED(exit_code)) + { + res = bpf_val_to_ring(data, __WTERMSIG(exit_code)); + } else { + res = bpf_val_to_ring(data, 0); + } + if (res != PPM_SUCCESS) + return res; + + /* Did it produce a core? */ + res = bpf_val_to_ring(data, __WCOREDUMP(exit_code) != 0); if (res != PPM_SUCCESS) return res; diff --git a/driver/event_table.c b/driver/event_table.c index 0962cd153..4d1ecec68 100644 --- a/driver/event_table.c +++ b/driver/event_table.c @@ -198,7 +198,7 @@ const struct ppm_event_info g_event_info[PPM_EVENT_MAX] = { /* PPME_SYSCALL_FORK_X */{"fork", EC_PROCESS, EF_MODIFIES_STATE | EF_OLD_VERSION, 16, {{"res", PT_PID, PF_DEC}, {"exe", PT_CHARBUF, PF_NA}, {"args", PT_BYTEBUF, PF_NA}, {"tid", PT_PID, PF_DEC}, {"pid", PT_PID, PF_DEC}, {"ptid", PT_PID, PF_DEC}, {"cwd", PT_CHARBUF, PF_NA}, {"fdlimit", PT_INT64, PF_DEC}, {"pgft_maj", PT_UINT64, PF_DEC}, {"pgft_min", PT_UINT64, PF_DEC}, {"vm_size", PT_UINT32, PF_DEC}, {"vm_rss", PT_UINT32, PF_DEC}, {"vm_swap", PT_UINT32, PF_DEC}, {"flags", PT_FLAGS32, PF_HEX, clone_flags}, {"uid", PT_UINT32, PF_DEC}, {"gid", PT_UINT32, PF_DEC} } }, /* PPME_SYSCALL_VFORK_E */{"vfork", EC_PROCESS, EF_MODIFIES_STATE | EF_OLD_VERSION, 0}, /* PPME_SYSCALL_VFORK_X */{"vfork", EC_PROCESS, EF_MODIFIES_STATE | EF_OLD_VERSION, 16, {{"res", PT_PID, PF_DEC}, {"exe", PT_CHARBUF, PF_NA}, {"args", PT_BYTEBUF, PF_NA}, {"tid", PT_PID, PF_DEC}, {"pid", PT_PID, PF_DEC}, {"ptid", PT_PID, PF_DEC}, {"cwd", PT_CHARBUF, PF_NA}, {"fdlimit", PT_INT64, PF_DEC}, {"pgft_maj", PT_UINT64, PF_DEC}, {"pgft_min", PT_UINT64, PF_DEC}, {"vm_size", PT_UINT32, PF_DEC}, {"vm_rss", PT_UINT32, PF_DEC}, {"vm_swap", PT_UINT32, PF_DEC}, {"flags", PT_FLAGS32, PF_HEX, clone_flags}, {"uid", PT_UINT32, PF_DEC}, {"gid", PT_UINT32, PF_DEC} } }, - /* PPME_PROCEXIT_1_E */{"procexit", EC_PROCESS, EF_MODIFIES_STATE, 1, {{"status", PT_ERRNO, PF_DEC} } }, + /* PPME_PROCEXIT_1_E */{"procexit", EC_PROCESS, EF_MODIFIES_STATE, 3, {{"status", PT_ERRNO, PF_DEC}, {"sig", PT_SIGTYPE, PF_DEC}, {"core", PT_UINT8, PF_DEC} } }, /* PPME_NA1 */{"NA1", EC_PROCESS, EF_UNUSED, 0}, /* PPME_SYSCALL_SENDFILE_E */{"sendfile", EC_IO_WRITE, EF_USES_FD | EF_DROP_SIMPLE_CONS, 4, {{"out_fd", PT_FD, PF_DEC}, {"in_fd", PT_FD, PF_DEC}, {"offset", PT_UINT64, PF_DEC}, {"size", PT_UINT64, PF_DEC} } }, /* PPME_SYSCALL_SENDFILE_X */{"sendfile", EC_IO_WRITE, EF_USES_FD | EF_DROP_SIMPLE_CONS, 2, {{"res", PT_ERRNO, PF_DEC}, {"offset", PT_UINT64, PF_DEC} } }, diff --git a/driver/ppm_fillers.c b/driver/ppm_fillers.c index 2fa487d84..6e28e9efb 100644 --- a/driver/ppm_fillers.c +++ b/driver/ppm_fillers.c @@ -95,6 +95,8 @@ or GPL2.txt for full copies of the license. #endif #include "kernel_hacks.h" +#include "systype_compat.h" + #endif /* UDIG */ #define merge_64(hi, lo) ((((unsigned long long)(hi)) << 32) + ((lo) & 0xffffffffUL)) @@ -4201,9 +4203,31 @@ int f_sys_procexit_e(struct event_filler_arguments *args) * status */ #ifndef UDIG - res = val_to_ring(args, args->sched_prev->exit_code, 0, false, 0); + /* Exit status */ + res = val_to_ring(args, __WEXITSTATUS(args->sched_prev->exit_code), 0, false, 0); + if (unlikely(res != PPM_SUCCESS)) + return res; + + /* If signaled -> signum, else 0 */ + if (__WIFSIGNALED(args->sched_prev->exit_code)) + { + res = val_to_ring(args, __WTERMSIG(args->sched_prev->exit_code), 0, false, 0); + } else { + res = val_to_ring(args, 0, 0, false, 0); + } + if (unlikely(res != PPM_SUCCESS)) + return res; + + /* Did it produce a core? */ + res = val_to_ring(args, __WCOREDUMP(args->sched_prev->exit_code) != 0, 0, false, 0); #else res = val_to_ring(args, 0, 0, false, 0); + if (unlikely(res != PPM_SUCCESS)) + return res; + res = val_to_ring(args, 0, 0, false, 0); + if (unlikely(res != PPM_SUCCESS)) + return res; + res = val_to_ring(args, 0, 0, false, 0); #endif if (unlikely(res != PPM_SUCCESS)) return res; diff --git a/driver/systype_compat.h b/driver/systype_compat.h new file mode 100644 index 000000000..f9560dd5c --- /dev/null +++ b/driver/systype_compat.h @@ -0,0 +1,14 @@ +/* If WIFEXITED(STATUS), the low-order 8 bits of the status. */ +#define __WEXITSTATUS(status) (((status) & 0xff00) >> 8) + +/* If WIFSIGNALED(STATUS), the terminating signal. */ +#define __WTERMSIG(status) ((status) & 0x7f) + +/* Nonzero if STATUS indicates termination by a signal. */ +#define __WIFSIGNALED(status) \ + (((signed char) (((status) & 0x7f) + 1) >> 1) > 0) + +/* Nonzero if STATUS indicates the child dumped core. */ +#define __WCOREDUMP(status) ((status) & __WCOREFLAG) + +#define __WCOREFLAG 0x80 \ No newline at end of file From 2258aba1b3f9e8f8b1a9e1af3a9f7a1eb6c1299c Mon Sep 17 00:00:00 2001 From: Federico Di Pierro Date: Fri, 12 Nov 2021 09:04:29 +0100 Subject: [PATCH 091/148] fix(driver): retain backward compatibility and add a new "ret" parameter to procexit, with user-seen exit code. Signed-off-by: Federico Di Pierro Co-authored-by: Mark Stemm --- driver/bpf/fillers.h | 7 ++++++- driver/event_table.c | 2 +- driver/ppm_fillers.c | 5 +++++ 3 files changed, 12 insertions(+), 2 deletions(-) diff --git a/driver/bpf/fillers.h b/driver/bpf/fillers.h index 0b2a8f227..08e80baed 100644 --- a/driver/bpf/fillers.h +++ b/driver/bpf/fillers.h @@ -3498,7 +3498,12 @@ FILLER(sys_procexit_e, false) exit_code = _READ(task->exit_code); - /* Real exit code */ + /* Exit status */ + res = bpf_val_to_ring(data, exit_code); + if (res != PPM_SUCCESS) + return res; + + /* Ret code */ res = bpf_val_to_ring(data, __WEXITSTATUS(exit_code)); if (res != PPM_SUCCESS) return res; diff --git a/driver/event_table.c b/driver/event_table.c index 4d1ecec68..d5f73ec24 100644 --- a/driver/event_table.c +++ b/driver/event_table.c @@ -198,7 +198,7 @@ const struct ppm_event_info g_event_info[PPM_EVENT_MAX] = { /* PPME_SYSCALL_FORK_X */{"fork", EC_PROCESS, EF_MODIFIES_STATE | EF_OLD_VERSION, 16, {{"res", PT_PID, PF_DEC}, {"exe", PT_CHARBUF, PF_NA}, {"args", PT_BYTEBUF, PF_NA}, {"tid", PT_PID, PF_DEC}, {"pid", PT_PID, PF_DEC}, {"ptid", PT_PID, PF_DEC}, {"cwd", PT_CHARBUF, PF_NA}, {"fdlimit", PT_INT64, PF_DEC}, {"pgft_maj", PT_UINT64, PF_DEC}, {"pgft_min", PT_UINT64, PF_DEC}, {"vm_size", PT_UINT32, PF_DEC}, {"vm_rss", PT_UINT32, PF_DEC}, {"vm_swap", PT_UINT32, PF_DEC}, {"flags", PT_FLAGS32, PF_HEX, clone_flags}, {"uid", PT_UINT32, PF_DEC}, {"gid", PT_UINT32, PF_DEC} } }, /* PPME_SYSCALL_VFORK_E */{"vfork", EC_PROCESS, EF_MODIFIES_STATE | EF_OLD_VERSION, 0}, /* PPME_SYSCALL_VFORK_X */{"vfork", EC_PROCESS, EF_MODIFIES_STATE | EF_OLD_VERSION, 16, {{"res", PT_PID, PF_DEC}, {"exe", PT_CHARBUF, PF_NA}, {"args", PT_BYTEBUF, PF_NA}, {"tid", PT_PID, PF_DEC}, {"pid", PT_PID, PF_DEC}, {"ptid", PT_PID, PF_DEC}, {"cwd", PT_CHARBUF, PF_NA}, {"fdlimit", PT_INT64, PF_DEC}, {"pgft_maj", PT_UINT64, PF_DEC}, {"pgft_min", PT_UINT64, PF_DEC}, {"vm_size", PT_UINT32, PF_DEC}, {"vm_rss", PT_UINT32, PF_DEC}, {"vm_swap", PT_UINT32, PF_DEC}, {"flags", PT_FLAGS32, PF_HEX, clone_flags}, {"uid", PT_UINT32, PF_DEC}, {"gid", PT_UINT32, PF_DEC} } }, - /* PPME_PROCEXIT_1_E */{"procexit", EC_PROCESS, EF_MODIFIES_STATE, 3, {{"status", PT_ERRNO, PF_DEC}, {"sig", PT_SIGTYPE, PF_DEC}, {"core", PT_UINT8, PF_DEC} } }, + /* PPME_PROCEXIT_1_E */{"procexit", EC_PROCESS, EF_MODIFIES_STATE, 4, {{"status", PT_ERRNO, PF_DEC}, {"ret", PT_ERRNO, PF_DEC}, {"sig", PT_SIGTYPE, PF_DEC}, {"core", PT_UINT8, PF_DEC} } }, /* PPME_NA1 */{"NA1", EC_PROCESS, EF_UNUSED, 0}, /* PPME_SYSCALL_SENDFILE_E */{"sendfile", EC_IO_WRITE, EF_USES_FD | EF_DROP_SIMPLE_CONS, 4, {{"out_fd", PT_FD, PF_DEC}, {"in_fd", PT_FD, PF_DEC}, {"offset", PT_UINT64, PF_DEC}, {"size", PT_UINT64, PF_DEC} } }, /* PPME_SYSCALL_SENDFILE_X */{"sendfile", EC_IO_WRITE, EF_USES_FD | EF_DROP_SIMPLE_CONS, 2, {{"res", PT_ERRNO, PF_DEC}, {"offset", PT_UINT64, PF_DEC} } }, diff --git a/driver/ppm_fillers.c b/driver/ppm_fillers.c index 6e28e9efb..4302e53d1 100644 --- a/driver/ppm_fillers.c +++ b/driver/ppm_fillers.c @@ -4204,6 +4204,11 @@ int f_sys_procexit_e(struct event_filler_arguments *args) */ #ifndef UDIG /* Exit status */ + res = val_to_ring(args, args->sched_prev->exit_code, 0, false, 0); + if (unlikely(res != PPM_SUCCESS)) + return res; + + /* Ret code */ res = val_to_ring(args, __WEXITSTATUS(args->sched_prev->exit_code), 0, false, 0); if (unlikely(res != PPM_SUCCESS)) return res; From 96320d10f03bf5d434369b6effd9dcff60aa0df2 Mon Sep 17 00:00:00 2001 From: Jason Dellaluce Date: Thu, 9 Sep 2021 14:50:14 +0000 Subject: [PATCH 092/148] update(driver): Add openat2 syscall Signed-off-by: Jason Dellaluce --- driver/bpf/fillers.h | 64 ++++++++++++++++++++++++++ driver/event_table.c | 2 + driver/fillers_table.c | 4 +- driver/flags_table.c | 10 ++++ driver/ppm_compat_unistd_32.h | 3 +- driver/ppm_events_public.h | 18 +++++++- driver/ppm_fillers.c | 62 +++++++++++++++++++++++++ driver/ppm_fillers.h | 1 + driver/ppm_flag_helpers.h | 25 ++++++++++ driver/syscall_table.c | 12 +++++ userspace/libscap/syscall_info_table.c | 1 + userspace/libsinsp/event.cpp | 3 +- userspace/libsinsp/examples/util.cpp | 3 +- userspace/libsinsp/filterchecks.cpp | 25 ++++++---- userspace/libsinsp/parsers.cpp | 7 +-- 15 files changed, 222 insertions(+), 18 deletions(-) diff --git a/driver/bpf/fillers.h b/driver/bpf/fillers.h index 08e80baed..a3d0c5b0d 100644 --- a/driver/bpf/fillers.h +++ b/driver/bpf/fillers.h @@ -2312,6 +2312,70 @@ FILLER(sys_openat_x, true) return res; } +FILLER(sys_openat2_x, true) +{ + unsigned long resolve; + unsigned long flags; + unsigned long val; + unsigned long mode; + long retval; + int res; + + retval = bpf_syscall_get_retval(data->ctx); + res = bpf_val_to_ring(data, retval); + if (res != PPM_SUCCESS) + return res; + + /* + * dirfd + */ + val = bpf_syscall_get_argument(data, 0); + if ((int)val == AT_FDCWD) + val = PPM_AT_FDCWD; + + res = bpf_val_to_ring(data, val); + if (res != PPM_SUCCESS) + return res; + + /* + * name + */ + val = bpf_syscall_get_argument(data, 1); + res = bpf_val_to_ring(data, val); + if (res != PPM_SUCCESS) + return res; + + /* + * how: we get the data structure, and put its fields in the buffer one by one + */ + val = bpf_syscall_get_argument(data, 2); + struct open_how *how = (struct open_how*) val; + + /* + * flags (extracted form how structure) + * Note that we convert them into the ppm portable representation before pushing them to the ring + */ + flags = open_flags_to_scap(how->flags); + res = bpf_val_to_ring(data, flags); + if (res != PPM_SUCCESS) + return res; + + /* + * mode (extracted form how structure) + */ + mode = open_modes_to_scap(how->flags, how->mode); + res = bpf_val_to_ring(data, mode); + if (res != PPM_SUCCESS) + return res; + + /* + * resolve (extracted form how structure) + */ + resolve = openat2_resolve_to_scap(how->resolve); + res = bpf_val_to_ring(data, resolve); + return res; +} + FILLER(sys_sendfile_e, true) { unsigned long val; diff --git a/driver/event_table.c b/driver/event_table.c index d5f73ec24..561fb0ae4 100644 --- a/driver/event_table.c +++ b/driver/event_table.c @@ -338,6 +338,8 @@ const struct ppm_event_info g_event_info[PPM_EVENT_MAX] = { /* PPME_NA1 */{"pluginevent", EC_OTHER, EF_UNUSED, 0}, /* PPME_CONTAINER_JSON_2_E */{"container", EC_PROCESS, EF_MODIFIES_STATE | EF_LARGE_PAYLOAD, 1, {{"json", PT_CHARBUF, PF_NA} } }, /* PPME_CONTAINER_JSON_2_X */{"container", EC_PROCESS, EF_UNUSED, 0}, + /* PPME_SYSCALL_OPENAT2_E */{"openat2", EC_FILE, EF_CREATES_FD | EF_MODIFIES_STATE, 0}, + /* PPME_SYSCALL_OPENAT2_X */{"openat2", EC_FILE, EF_CREATES_FD | EF_MODIFIES_STATE, 6, {{"fd", PT_FD, PF_DEC}, {"dirfd", PT_FD, PF_DEC}, {"name", PT_FSRELPATH, PF_NA, DIRFD_PARAM(1)}, {"flags", PT_FLAGS32, PF_HEX, file_flags}, {"mode", PT_UINT32, PF_OCT}, {"resolve", PT_FLAGS32, PF_HEX, openat2_flags} } }, /* NB: Starting from scap version 1.2, event types will no longer be changed when an event is modified, and the only kind of change permitted for pre-existent events is adding parameters. * New event types are allowed only for new syscalls or new internal events. * The number of parameters can be used to differentiate between event versions. diff --git a/driver/fillers_table.c b/driver/fillers_table.c index 094531e6f..88c5d824b 100644 --- a/driver/fillers_table.c +++ b/driver/fillers_table.c @@ -306,6 +306,8 @@ const struct ppm_event_entry g_ppm_events[PPM_EVENT_MAX] = { [PPME_SYSCALL_RENAMEAT2_E] = {FILLER_REF(sys_empty)}, [PPME_SYSCALL_RENAMEAT2_X] = {FILLER_REF(sys_renameat2_x)}, [PPME_SYSCALL_USERFAULTFD_E] = {FILLER_REF(sys_empty)}, - [PPME_SYSCALL_USERFAULTFD_X] = {FILLER_REF(sys_autofill), 2, APT_REG, {{AF_ID_RETVAL}, {0} } } + [PPME_SYSCALL_USERFAULTFD_X] = {FILLER_REF(sys_autofill), 2, APT_REG, {{AF_ID_RETVAL}, {0} } }, + [PPME_SYSCALL_OPENAT2_E] = {FILLER_REF(sys_empty)}, + [PPME_SYSCALL_OPENAT2_X] = {FILLER_REF(sys_openat2_x)}, #endif /* WDIG */ }; diff --git a/driver/flags_table.c b/driver/flags_table.c index 725903dc3..284f12c6c 100644 --- a/driver/flags_table.c +++ b/driver/flags_table.c @@ -525,3 +525,13 @@ const struct ppm_name_value renameat2_flags[] = { {"RENAME_WHITEOUT", PPM_RENAME_WHITEOUT}, {0, 0}, }; + +const struct ppm_name_value openat2_flags[] = { + {"RESOLVE_BENEATH", PPM_RESOLVE_BENEATH}, + {"RESOLVE_IN_ROOT", PPM_RESOLVE_IN_ROOT}, + {"RESOLVE_NO_MAGICLINKS", PPM_RESOLVE_NO_MAGICLINKS}, + {"RESOLVE_NO_SYMLINKS", PPM_RESOLVE_NO_SYMLINKS}, + {"RESOLVE_NO_XDEV", PPM_RESOLVE_NO_XDEV}, + {"RESOLVE_CACHED", PPM_RESOLVE_CACHED}, + {0, 0}, +}; diff --git a/driver/ppm_compat_unistd_32.h b/driver/ppm_compat_unistd_32.h index 86b404611..63c1e3f0e 100644 --- a/driver/ppm_compat_unistd_32.h +++ b/driver/ppm_compat_unistd_32.h @@ -356,10 +356,11 @@ #define __NR_ia32_process_vm_writev 348 #define __NR_ia32_renameat2 349 #define __NR_ia32_userfaultfd 350 +#define __NR_ia32_openat2 351 #ifdef __KERNEL__ -#define NR_ia32_syscalls 351 +#define NR_ia32_syscalls 352 #define __ARCH_WANT_IPC_PARSE_VERSION #define __ARCH_WANT_OLD_READDIR diff --git a/driver/ppm_events_public.h b/driver/ppm_events_public.h index 56bd2871e..c403a43d6 100644 --- a/driver/ppm_events_public.h +++ b/driver/ppm_events_public.h @@ -595,6 +595,16 @@ or GPL2.txt for full copies of the license. #define PPM_RENAME_EXCHANGE (1 << 1) /* Exchange source and dest */ #define PPM_RENAME_WHITEOUT (1 << 2) /* Whiteout source */ +/* + * Openat2 resolve flags + */ +#define PPM_RESOLVE_BENEATH (1 << 0) +#define PPM_RESOLVE_IN_ROOT (1 << 1) +#define PPM_RESOLVE_NO_MAGICLINKS (1 << 2) +#define PPM_RESOLVE_NO_SYMLINKS (1 << 3) +#define PPM_RESOLVE_NO_XDEV (1 << 4) +#define PPM_RESOLVE_CACHED (1 << 5) + /* * SuS says limits have to be unsigned. * Which makes a ton more sense anyway. @@ -965,7 +975,9 @@ enum ppm_event_type { PPME_PLUGINEVENT_X = 323, PPME_CONTAINER_JSON_2_E = 324, PPME_CONTAINER_JSON_2_X = 325, - PPM_EVENT_MAX = 326 + PPME_SYSCALL_OPENAT2_E = 326, + PPME_SYSCALL_OPENAT2_X = 327, + PPM_EVENT_MAX = 328 }; /*@}*/ @@ -1295,7 +1307,8 @@ enum ppm_syscall_code { PPM_SC_FADVISE64 = 319, PPM_SC_RENAMEAT2 = 320, PPM_SC_USERFAULTFD = 321, - PPM_SC_MAX = 322, + PPM_SC_OPENAT2 = 322, + PPM_SC_MAX = 323, }; /* @@ -1523,6 +1536,7 @@ extern const struct ppm_name_value unlinkat_flags[]; extern const struct ppm_name_value linkat_flags[]; extern const struct ppm_name_value chmod_mode[]; extern const struct ppm_name_value renameat2_flags[]; +extern const struct ppm_name_value openat2_flags[]; extern const struct ppm_param_info sockopt_dynamic_param[]; extern const struct ppm_param_info ptrace_dynamic_param[]; diff --git a/driver/ppm_fillers.c b/driver/ppm_fillers.c index 4302e53d1..389dc1346 100644 --- a/driver/ppm_fillers.c +++ b/driver/ppm_fillers.c @@ -4186,6 +4186,68 @@ int f_sys_symlinkat_x(struct event_filler_arguments *args) return add_sentinel(args); } + +int f_sys_openat2_x(struct event_filler_arguments *args) +{ + unsigned long val; + int res; + int64_t retval; + + retval = (int64_t)syscall_get_return_value(current, args->regs); + res = val_to_ring(args, retval, 0, false, 0); + if (unlikely(res != PPM_SUCCESS)) + return res; + + /* + * dirfd + */ + syscall_get_arguments_deprecated(current, args->regs, 0, 1, &val); + + if ((int)val == AT_FDCWD) + val = PPM_AT_FDCWD; + + res = val_to_ring(args, val, 0, false, 0); + if (unlikely(res != PPM_SUCCESS)) + return res; + + /* + * name + */ + syscall_get_arguments_deprecated(current, args->regs, 1, 1, &val); + res = val_to_ring(args, val, 0, true, 0); + if (unlikely(res != PPM_SUCCESS)) + return res; + + /* + * how: we get the data structure, and put its fields in the buffer one by one + */ + syscall_get_arguments_deprecated(current, args->regs, 2, 1, &val); + struct open_how *how = (struct open_how*) val; + + /* + * flags (extracted form how structure) + * Note that we convert them into the ppm portable representation before pushing them to the ring + */ + res = val_to_ring(args, open_flags_to_scap(how->flags), 0, true, 0); + if (unlikely(res != PPM_SUCCESS)) + return res; + + /* + * mode (extracted form how structure) + */ + res = val_to_ring(args, open_modes_to_scap(how->flags, how->mode), 0, true, 0); + if (unlikely(res != PPM_SUCCESS)) + return res; + + /* + * resolve (extracted form how structure) + */ + res = val_to_ring(args, openat2_resolve_to_scap(how->resolve), 0, true, 0); + if (unlikely(res != PPM_SUCCESS)) + return res; + + return add_sentinel(args); +} #endif /* WDIG */ int f_sys_procexit_e(struct event_filler_arguments *args) diff --git a/driver/ppm_fillers.h b/driver/ppm_fillers.h index be03cc3f4..94567aa6b 100644 --- a/driver/ppm_fillers.h +++ b/driver/ppm_fillers.h @@ -114,6 +114,7 @@ or GPL2.txt for full copies of the license. FN(sys_fchmod_x) \ FN(sys_mkdirat_x) \ FN(sys_openat_x) \ + FN(sys_openat2_x) \ FN(sys_linkat_x) \ FN(terminate_filler) diff --git a/driver/ppm_flag_helpers.h b/driver/ppm_flag_helpers.h index 03ef765b0..d17c00b79 100644 --- a/driver/ppm_flag_helpers.h +++ b/driver/ppm_flag_helpers.h @@ -161,6 +161,31 @@ static __always_inline u32 open_modes_to_scap(unsigned long flags, res |= PPM_S_ISVTX; return res; +} + +static __always_inline u32 openat2_resolve_to_scap(unsigned long flags) +{ + u32 res = 0; + #ifdef _LINUX_OPENAT2_H + if (flags & RESOLVE_NO_XDEV) + res |= PPM_RESOLVE_NO_XDEV; + + if (flags & RESOLVE_NO_MAGICLINKS) + res |= PPM_RESOLVE_NO_MAGICLINKS; + + if (flags & RESOLVE_NO_SYMLINKS) + res |= PPM_RESOLVE_NO_SYMLINKS; + + if (flags & RESOLVE_BENEATH) + res |= PPM_RESOLVE_BENEATH; + + if (flags & RESOLVE_IN_ROOT) + res |= PPM_RESOLVE_IN_ROOT; + + if (flags & RESOLVE_CACHED) + res |= PPM_RESOLVE_CACHED; + #endif + return res; #endif // WDIG } diff --git a/driver/syscall_table.c b/driver/syscall_table.c index 81f408893..0d4af9a8f 100644 --- a/driver/syscall_table.c +++ b/driver/syscall_table.c @@ -358,6 +358,9 @@ const struct syscall_evt_pair g_syscall_table[SYSCALL_TABLE_SIZE] = { #ifdef __NR_userfaultfd [__NR_userfaultfd - SYSCALL_TABLE_ID0] = {UF_USED | UF_NEVER_DROP, PPME_SYSCALL_USERFAULTFD_E, PPME_SYSCALL_USERFAULTFD_X}, #endif +#ifdef __NR_openat2 + [__NR_openat2 - SYSCALL_TABLE_ID0] = {UF_USED, PPME_SYSCALL_OPENAT2_E, PPME_SYSCALL_OPENAT2_X}, +#endif }; /* @@ -977,6 +980,9 @@ const enum ppm_syscall_code g_syscall_code_routing_table[SYSCALL_TABLE_SIZE] = { #ifdef __NR_userfaultfd [__NR_userfaultfd - SYSCALL_TABLE_ID0] = PPM_SC_USERFAULTFD, #endif +#ifdef __NR_openat2 + [__NR_openat2 - SYSCALL_TABLE_ID0] = PPM_SC_OPENAT2, +#endif }; #ifdef CONFIG_IA32_EMULATION @@ -1216,6 +1222,9 @@ const struct syscall_evt_pair g_syscall_ia32_table[SYSCALL_TABLE_SIZE] = { #ifdef __NR_ia32_userfaultfd [__NR_ia32_userfaultfd - SYSCALL_TABLE_ID0] = {UF_USED | UF_NEVER_DROP, PPME_SYSCALL_USERFAULTFD_E, PPME_SYSCALL_USERFAULTFD_X}, #endif +#ifdef __NR_ia32_openat2 + [__NR_ia32_openat2 - SYSCALL_TABLE_ID0] = {UF_USED, PPME_SYSCALL_OPENAT2_E, PPME_SYSCALL_OPENAT2_X}, +#endif }; /* @@ -1775,6 +1784,9 @@ const enum ppm_syscall_code g_syscall_ia32_code_routing_table[SYSCALL_TABLE_SIZE #ifdef __NR_ia32_userfaultfd [__NR_ia32_userfaultfd - SYSCALL_TABLE_ID0] = PPM_SC_USERFAULTFD, #endif +#ifdef __NR_ia32_openat2 + [__NR_ia32_openat2 - SYSCALL_TABLE_ID0] = PPM_SC_OPENAT2, +#endif }; #endif /* CONFIG_IA32_EMULATION */ diff --git a/userspace/libscap/syscall_info_table.c b/userspace/libscap/syscall_info_table.c index 2ade0b070..436794e54 100644 --- a/userspace/libscap/syscall_info_table.c +++ b/userspace/libscap/syscall_info_table.c @@ -351,6 +351,7 @@ const struct ppm_syscall_desc g_syscall_info_table[PPM_SC_MAX] = { /*PPM_SC_FADVISE64*/ { EC_IO_OTHER, (enum ppm_event_flags)(EF_NONE), "fadvise64" }, /*PPM_SC_RENAMEAT2*/ { EC_FILE, (enum ppm_event_flags)(EF_NONE), "renameat2" }, /*PPM_SC_USERFAULTFD*/ { EC_FILE, (enum ppm_event_flags)(EF_NONE), "userfaultfd" }, + /*PPM_SC_OPENAT2*/ { EC_FILE, (enum ppm_event_flags)(EF_NONE), "openat2" }, }; bool validate_info_table_size() diff --git a/userspace/libsinsp/event.cpp b/userspace/libsinsp/event.cpp index 749c42cc0..9c95a36c8 100644 --- a/userspace/libsinsp/event.cpp +++ b/userspace/libsinsp/event.cpp @@ -2669,7 +2669,8 @@ bool sinsp_evt::is_file_open_error() const ((m_pevt->type == PPME_SYSCALL_OPEN_X) || (m_pevt->type == PPME_SYSCALL_CREAT_X) || (m_pevt->type == PPME_SYSCALL_OPENAT_X) || - (m_pevt->type == PPME_SYSCALL_OPENAT_2_X)); + (m_pevt->type == PPME_SYSCALL_OPENAT_2_X) || + (m_pevt->type == PPME_SYSCALL_OPENAT2_X)); } bool sinsp_evt::is_file_error() const diff --git a/userspace/libsinsp/examples/util.cpp b/userspace/libsinsp/examples/util.cpp index 3ebe86f71..581520c50 100644 --- a/userspace/libsinsp/examples/util.cpp +++ b/userspace/libsinsp/examples/util.cpp @@ -132,7 +132,8 @@ std::string get_event_type(uint16_t type) case PPME_SYSCALL_OPENAT_E: case PPME_SYSCALL_OPENAT_2_E: case PPME_SYSCALL_OPENAT_X: - case PPME_SYSCALL_OPENAT_2_X: return "openat"; + case PPME_SYSCALL_OPENAT_2_X: + case PPME_SYSCALL_OPENAT2_X: return "openat"; case PPME_SYSCALL_PIPE_E: case PPME_SYSCALL_PIPE_X: return "pipe"; case PPME_SYSCALL_POLL_E: diff --git a/userspace/libsinsp/filterchecks.cpp b/userspace/libsinsp/filterchecks.cpp index 23dec76b7..6da11964c 100644 --- a/userspace/libsinsp/filterchecks.cpp +++ b/userspace/libsinsp/filterchecks.cpp @@ -204,6 +204,7 @@ bool sinsp_filter_check_fd::extract_fdname_from_creator(sinsp_evt *evt, OUT uint } case PPME_SYSCALL_OPENAT_X: case PPME_SYSCALL_OPENAT_2_X: + case PPME_SYSCALL_OPENAT2_X: { sinsp_evt enter_evt; sinsp_evt_param *parinfo; @@ -328,7 +329,8 @@ uint8_t* sinsp_filter_check_fd::extract_from_null_fd(sinsp_evt *evt, OUT uint32_ case TYPE_FILENAME: { if(evt->get_type() != PPME_SYSCALL_OPEN_E && evt->get_type() != PPME_SYSCALL_OPENAT_E && - evt->get_type() != PPME_SYSCALL_OPENAT_2_E && evt->get_type() != PPME_SYSCALL_CREAT_E) + evt->get_type() != PPME_SYSCALL_OPENAT_2_E && evt->get_type() != PPME_SYSCALL_OPENAT2_E && + evt->get_type() != PPME_SYSCALL_CREAT_E) { return NULL; } @@ -363,6 +365,7 @@ uint8_t* sinsp_filter_check_fd::extract_from_null_fd(sinsp_evt *evt, OUT uint32_ case PPME_SYSCALL_OPEN_E: case PPME_SYSCALL_OPENAT_E: case PPME_SYSCALL_OPENAT_2_E: + case PPME_SYSCALL_OPENAT2_E: case PPME_SYSCALL_CREAT_E: m_tcstr[0] = CHAR_FD_FILE; m_tcstr[1] = 0; @@ -2952,13 +2955,13 @@ const filtercheck_field_info sinsp_filter_check_event_fields[] = {PT_UINT64, EPF_TABLE_ONLY, PF_DEC, "evt.buflen.net", "the length of the binary data buffer, but only for network I/O events."}, {PT_UINT64, EPF_TABLE_ONLY, PF_DEC, "evt.buflen.net.in", "the length of the binary data buffer, but only for input network I/O events."}, {PT_UINT64, EPF_TABLE_ONLY, PF_DEC, "evt.buflen.net.out", "the length of the binary data buffer, but only for output network I/O events."}, - {PT_BOOL, EPF_NONE, PF_NA, "evt.is_open_read", "'true' for open/openat events where the path was opened for reading"}, - {PT_BOOL, EPF_NONE, PF_NA, "evt.is_open_write", "'true' for open/openat events where the path was opened for writing"}, + {PT_BOOL, EPF_NONE, PF_NA, "evt.is_open_read", "'true' for open/openat/openat2 events where the path was opened for reading"}, + {PT_BOOL, EPF_NONE, PF_NA, "evt.is_open_write", "'true' for open/openat/openat2 events where the path was opened for writing"}, {PT_CHARBUF, EPF_TABLE_ONLY, PF_NA, "evt.infra.docker.name", "for docker infrastructure events, the name of the event."}, {PT_CHARBUF, EPF_TABLE_ONLY, PF_NA, "evt.infra.docker.container.id", "for docker infrastructure events, the id of the impacted container."}, {PT_CHARBUF, EPF_TABLE_ONLY, PF_NA, "evt.infra.docker.container.name", "for docker infrastructure events, the name of the impacted container."}, {PT_CHARBUF, EPF_TABLE_ONLY, PF_NA, "evt.infra.docker.container.image", "for docker infrastructure events, the image name of the impacted container."}, - {PT_BOOL, EPF_NONE, PF_NA, "evt.is_open_exec", "'true' for open/openat or creat events where a file is created with execute permissions"}, + {PT_BOOL, EPF_NONE, PF_NA, "evt.is_open_exec", "'true' for open/openat/openat2 or creat events where a file is created with execute permissions"}, }; sinsp_filter_check_event::sinsp_filter_check_event() @@ -3301,7 +3304,7 @@ uint8_t *sinsp_filter_check_event::extract_abspath(sinsp_evt *evt, OUT uint32_t dirfdarg = "linkdirfd"; patharg = "linkpath"; } - else if(etype == PPME_SYSCALL_OPENAT_E || etype == PPME_SYSCALL_OPENAT_2_X) + else if(etype == PPME_SYSCALL_OPENAT_E || etype == PPME_SYSCALL_OPENAT_2_X || etype == PPME_SYSCALL_OPENAT2_X) { dirfdarg = "dirfd"; patharg = "name"; @@ -4231,7 +4234,8 @@ uint8_t* sinsp_filter_check_event::extract(sinsp_evt *evt, OUT uint32_t* len, bo if(etype == PPME_SYSCALL_OPEN_X || etype == PPME_SYSCALL_CREAT_X || etype == PPME_SYSCALL_OPENAT_X || - etype == PPME_SYSCALL_OPENAT_2_X) + etype == PPME_SYSCALL_OPENAT_2_X || + etype == PPME_SYSCALL_OPENAT2_X) { return extract_error_count(evt, len); } @@ -4307,6 +4311,7 @@ uint8_t* sinsp_filter_check_event::extract(sinsp_evt *evt, OUT uint32_t* len, bo etype == PPME_SYSCALL_CREAT_X || etype == PPME_SYSCALL_OPENAT_X || etype == PPME_SYSCALL_OPENAT_2_X || + etype == PPME_SYSCALL_OPENAT2_X || etype == PPME_SOCKET_ACCEPT_X || etype == PPME_SOCKET_ACCEPT_5_X || etype == PPME_SOCKET_ACCEPT4_X || @@ -4454,11 +4459,13 @@ uint8_t* sinsp_filter_check_event::extract(sinsp_evt *evt, OUT uint32_t* len, bo if(etype == PPME_SYSCALL_OPEN_X || etype == PPME_SYSCALL_OPENAT_E || - etype == PPME_SYSCALL_OPENAT_2_X) + etype == PPME_SYSCALL_OPENAT_2_X || + etype == PPME_SYSCALL_OPENAT2_X) { + bool is_new_version = etype == PPME_SYSCALL_OPENAT_2_X || etype == PPME_SYSCALL_OPENAT2_X; // For both OPEN_X and OPENAT_E, // flags is the 3rd argument. - parinfo = evt->get_param(etype == PPME_SYSCALL_OPENAT_2_X ? 3 : 2); + parinfo = evt->get_param(is_new_version ? 3 : 2); ASSERT(parinfo->m_len == sizeof(uint32_t)); uint32_t flags = *(uint32_t *)parinfo->m_val; @@ -4479,7 +4486,7 @@ uint8_t* sinsp_filter_check_event::extract(sinsp_evt *evt, OUT uint32_t* len, bo if(m_field_id == TYPE_ISOPEN_EXEC && (flags & (PPM_O_TMPFILE | PPM_O_CREAT))) { - parinfo = evt->get_param(etype == PPME_SYSCALL_OPENAT_2_X ? 4 : 3); + parinfo = evt->get_param(is_new_version ? 4 : 3); ASSERT(parinfo->m_len == sizeof(uint32_t)); uint32_t mode_bits = *(uint32_t *)parinfo->m_val; m_u32val = (mode_bits & is_exec_mask)? 1 : 0; diff --git a/userspace/libsinsp/parsers.cpp b/userspace/libsinsp/parsers.cpp index 3202b0672..1e7b4b470 100644 --- a/userspace/libsinsp/parsers.cpp +++ b/userspace/libsinsp/parsers.cpp @@ -309,6 +309,7 @@ void sinsp_parser::process_event(sinsp_evt *evt) case PPME_SYSCALL_CREAT_X: case PPME_SYSCALL_OPENAT_X: case PPME_SYSCALL_OPENAT_2_X: + case PPME_SYSCALL_OPENAT2_X: parse_open_openat_creat_exit(evt); break; case PPME_SYSCALL_SELECT_E: @@ -2037,7 +2038,7 @@ void sinsp_parser::parse_open_openat_creat_exit(sinsp_evt *evt) } - if(etype != PPME_SYSCALL_OPENAT_2_X) + if(etype != PPME_SYSCALL_OPENAT_2_X && etype != PPME_SYSCALL_OPENAT2_X) { // // Load the enter event so we can access its arguments @@ -2110,7 +2111,7 @@ void sinsp_parser::parse_open_openat_creat_exit(sinsp_evt *evt) parse_openat_dir(evt, name, dirfd, &sdir); } - else if(etype == PPME_SYSCALL_OPENAT_2_X) + else if(etype == PPME_SYSCALL_OPENAT_2_X || etype == PPME_SYSCALL_OPENAT2_X) { parinfo = evt->get_param(2); name = parinfo->m_val; @@ -2124,7 +2125,7 @@ void sinsp_parser::parse_open_openat_creat_exit(sinsp_evt *evt) ASSERT(parinfo->m_len == sizeof(int64_t)); int64_t dirfd = *(int64_t *)parinfo->m_val; - if(evt->get_num_params() > 5) + if(etype == PPME_SYSCALL_OPENAT_2_X && evt->get_num_params() > 5) { parinfo = evt->get_param(5); ASSERT(parinfo->m_len == sizeof(uint32_t)); From 17e55b201901c28f42390787c88b9373bfbf1b56 Mon Sep 17 00:00:00 2001 From: Jason Dellaluce Date: Fri, 10 Sep 2021 12:50:55 +0000 Subject: [PATCH 093/148] fix(driver): fix driver issues for openat2 Signed-off-by: Jason Dellaluce Co-authored-by: Federico Di Pierro --- driver/bpf/fillers.h | 27 +++++++++++++++++++------- driver/ppm_fillers.c | 36 +++++++++++++++++++++++++++-------- driver/ppm_flag_helpers.h | 40 ++++++++++++++++++++++++--------------- 3 files changed, 73 insertions(+), 30 deletions(-) diff --git a/driver/bpf/fillers.h b/driver/bpf/fillers.h index a3d0c5b0d..0fa67cfb9 100644 --- a/driver/bpf/fillers.h +++ b/driver/bpf/fillers.h @@ -2320,6 +2320,9 @@ FILLER(sys_openat2_x, true) unsigned long mode; long retval; int res; +#if LINUX_VERSION_CODE >= KERNEL_VERSION(5, 6, 0) + struct open_how how; +#endif retval = bpf_syscall_get_retval(data->ctx); res = bpf_val_to_ring(data, retval); @@ -2345,33 +2348,43 @@ FILLER(sys_openat2_x, true) if (res != PPM_SUCCESS) return res; +#if LINUX_VERSION_CODE >= KERNEL_VERSION(5, 6, 0) /* * how: we get the data structure, and put its fields in the buffer one by one */ val = bpf_syscall_get_argument(data, 2); - struct open_how *how = (struct open_how*) val; + if (bpf_probe_read(&how, sizeof(struct open_how), (void *)val)) { + return PPM_FAILURE_INVALID_USER_MEMORY; + } + flags = open_flags_to_scap(how.flags); + mode = open_modes_to_scap(how.flags, how.mode); + resolve = openat2_resolve_to_scap(how.resolve); +#else + flags = 0; + mode = 0; + resolve = 0; +#endif /* - * flags (extracted form how structure) + * flags (extracted from open_how structure) * Note that we convert them into the ppm portable representation before pushing them to the ring */ - flags = open_flags_to_scap(how->flags); res = bpf_val_to_ring(data, flags); if (res != PPM_SUCCESS) return res; /* - * mode (extracted form how structure) + * mode (extracted from open_how structure) + * Note that we convert them into the ppm portable representation before pushing them to the ring */ - mode = open_modes_to_scap(how->flags, how->mode); res = bpf_val_to_ring(data, mode); if (res != PPM_SUCCESS) return res; /* - * resolve (extracted form how structure) + * resolve (extracted from open_how structure) + * Note that we convert them into the ppm portable representation before pushing them to the ring */ - resolve = openat2_resolve_to_scap(how->resolve); res = bpf_val_to_ring(data, resolve); return res; } diff --git a/driver/ppm_fillers.c b/driver/ppm_fillers.c index 389dc1346..f19875608 100644 --- a/driver/ppm_fillers.c +++ b/driver/ppm_fillers.c @@ -4189,9 +4189,15 @@ int f_sys_symlinkat_x(struct event_filler_arguments *args) int f_sys_openat2_x(struct event_filler_arguments *args) { + unsigned long resolve; + unsigned long flags; unsigned long val; + unsigned long mode; int res; int64_t retval; +#if LINUX_VERSION_CODE >= KERNEL_VERSION(5, 6, 0) + struct open_how how; +#endif retval = (int64_t)syscall_get_return_value(current, args->regs); res = val_to_ring(args, retval, 0, false, 0); @@ -4218,31 +4224,45 @@ int f_sys_openat2_x(struct event_filler_arguments *args) if (unlikely(res != PPM_SUCCESS)) return res; + +#if LINUX_VERSION_CODE >= KERNEL_VERSION(5, 6, 0) /* * how: we get the data structure, and put its fields in the buffer one by one */ syscall_get_arguments_deprecated(current, args->regs, 2, 1, &val); - struct open_how *how = (struct open_how*) val; - + res = ppm_copy_from_user(&how, (void *)val, sizeof(struct open_how)); + if (unlikely(res != 0)) + return PPM_FAILURE_INVALID_USER_MEMORY; + + flags = open_flags_to_scap(how.flags); + mode = open_modes_to_scap(how.flags, how.mode); + resolve = openat2_resolve_to_scap(how.resolve); +#else + flags = 0; + mode = 0; + resolve = 0; +#endif /* - * flags (extracted form how structure) + * flags (extracted from open_how structure) * Note that we convert them into the ppm portable representation before pushing them to the ring */ - res = val_to_ring(args, open_flags_to_scap(how->flags), 0, true, 0); + res = val_to_ring(args, flags, 0, true, 0); if (unlikely(res != PPM_SUCCESS)) return res; /* - * mode (extracted form how structure) + * mode (extracted from open_how structure) + * Note that we convert them into the ppm portable representation before pushing them to the ring */ - res = val_to_ring(args, open_modes_to_scap(how->flags, how->mode), 0, true, 0); + res = val_to_ring(args, mode, 0, true, 0); if (unlikely(res != PPM_SUCCESS)) return res; /* - * resolve (extracted form how structure) + * resolve (extracted from open_how structure) + * Note that we convert them into the ppm portable representation before pushing them to the ring */ - res = val_to_ring(args, openat2_resolve_to_scap(how->resolve), 0, true, 0); + res = val_to_ring(args, resolve, 0, true, 0); if (unlikely(res != PPM_SUCCESS)) return res; diff --git a/driver/ppm_flag_helpers.h b/driver/ppm_flag_helpers.h index d17c00b79..1a99cae7f 100644 --- a/driver/ppm_flag_helpers.h +++ b/driver/ppm_flag_helpers.h @@ -166,25 +166,35 @@ static __always_inline u32 open_modes_to_scap(unsigned long flags, static __always_inline u32 openat2_resolve_to_scap(unsigned long flags) { u32 res = 0; - #ifdef _LINUX_OPENAT2_H - if (flags & RESOLVE_NO_XDEV) - res |= PPM_RESOLVE_NO_XDEV; +#ifdef RESOLVE_NO_XDEV + if (flags & RESOLVE_NO_XDEV) + res |= PPM_RESOLVE_NO_XDEV; +#endif - if (flags & RESOLVE_NO_MAGICLINKS) - res |= PPM_RESOLVE_NO_MAGICLINKS; +#ifdef RESOLVE_NO_MAGICLINKS + if (flags & RESOLVE_NO_MAGICLINKS) + res |= PPM_RESOLVE_NO_MAGICLINKS; +#endif - if (flags & RESOLVE_NO_SYMLINKS) - res |= PPM_RESOLVE_NO_SYMLINKS; +#ifdef RESOLVE_NO_SYMLINKS + if (flags & RESOLVE_NO_SYMLINKS) + res |= PPM_RESOLVE_NO_SYMLINKS; +#endif - if (flags & RESOLVE_BENEATH) - res |= PPM_RESOLVE_BENEATH; +#ifdef RESOLVE_BENEATH + if (flags & RESOLVE_BENEATH) + res |= PPM_RESOLVE_BENEATH; +#endif - if (flags & RESOLVE_IN_ROOT) - res |= PPM_RESOLVE_IN_ROOT; - - if (flags & RESOLVE_CACHED) - res |= PPM_RESOLVE_CACHED; - #endif +#ifdef RESOLVE_IN_ROOT + if (flags & RESOLVE_IN_ROOT) + res |= PPM_RESOLVE_IN_ROOT; +#endif + +#ifdef RESOLVE_CACHED + if (flags & RESOLVE_CACHED) + res |= PPM_RESOLVE_CACHED; +#endif return res; #endif // WDIG } From 1ed3e2a15dad1347459f1d55838bbbb8ae352266 Mon Sep 17 00:00:00 2001 From: Jason Dellaluce Date: Tue, 14 Sep 2021 16:24:49 +0000 Subject: [PATCH 094/148] update(driver): minor changes on openat2 support Signed-off-by: Jason Dellaluce --- driver/bpf/fillers.h | 4 ++-- driver/ppm_fillers.c | 4 ++-- userspace/libsinsp/examples/util.cpp | 4 ++-- 3 files changed, 6 insertions(+), 6 deletions(-) diff --git a/driver/bpf/fillers.h b/driver/bpf/fillers.h index 0fa67cfb9..c3448ea70 100644 --- a/driver/bpf/fillers.h +++ b/driver/bpf/fillers.h @@ -2320,7 +2320,7 @@ FILLER(sys_openat2_x, true) unsigned long mode; long retval; int res; -#if LINUX_VERSION_CODE >= KERNEL_VERSION(5, 6, 0) +#ifdef __NR_openat2 struct open_how how; #endif @@ -2348,7 +2348,7 @@ FILLER(sys_openat2_x, true) if (res != PPM_SUCCESS) return res; -#if LINUX_VERSION_CODE >= KERNEL_VERSION(5, 6, 0) +#ifdef __NR_openat2 /* * how: we get the data structure, and put its fields in the buffer one by one */ diff --git a/driver/ppm_fillers.c b/driver/ppm_fillers.c index f19875608..901b7069d 100644 --- a/driver/ppm_fillers.c +++ b/driver/ppm_fillers.c @@ -4195,7 +4195,7 @@ int f_sys_openat2_x(struct event_filler_arguments *args) unsigned long mode; int res; int64_t retval; -#if LINUX_VERSION_CODE >= KERNEL_VERSION(5, 6, 0) +#ifdef __NR_openat2 struct open_how how; #endif @@ -4225,7 +4225,7 @@ int f_sys_openat2_x(struct event_filler_arguments *args) return res; -#if LINUX_VERSION_CODE >= KERNEL_VERSION(5, 6, 0) +#ifdef __NR_openat2 /* * how: we get the data structure, and put its fields in the buffer one by one */ diff --git a/userspace/libsinsp/examples/util.cpp b/userspace/libsinsp/examples/util.cpp index 581520c50..104fae20d 100644 --- a/userspace/libsinsp/examples/util.cpp +++ b/userspace/libsinsp/examples/util.cpp @@ -132,8 +132,8 @@ std::string get_event_type(uint16_t type) case PPME_SYSCALL_OPENAT_E: case PPME_SYSCALL_OPENAT_2_E: case PPME_SYSCALL_OPENAT_X: - case PPME_SYSCALL_OPENAT_2_X: - case PPME_SYSCALL_OPENAT2_X: return "openat"; + case PPME_SYSCALL_OPENAT_2_X: return "openat"; + case PPME_SYSCALL_OPENAT2_X: return "openat2"; case PPME_SYSCALL_PIPE_E: case PPME_SYSCALL_PIPE_X: return "pipe"; case PPME_SYSCALL_POLL_E: From 941b051e0529bc2cfe0ec87214b3c6aeea87500b Mon Sep 17 00:00:00 2001 From: Federico Di Pierro Date: Wed, 13 Oct 2021 10:29:45 +0200 Subject: [PATCH 095/148] fix(driver/bpf): use bpf_ktime_get_boot_ns() where available to fix eBPF events timestamp, fallbacking at current bpf_ktime_get_ns() where not available. Signed-off-by: Federico Di Pierro --- driver/bpf/bpf_helpers.h | 11 ++++++++++- driver/bpf/plumbing_helpers.h | 2 +- 2 files changed, 11 insertions(+), 2 deletions(-) diff --git a/driver/bpf/bpf_helpers.h b/driver/bpf/bpf_helpers.h index 1a25b9214..abfa41bf6 100644 --- a/driver/bpf/bpf_helpers.h +++ b/driver/bpf/bpf_helpers.h @@ -18,8 +18,17 @@ static int (*bpf_map_delete_elem)(void *map, void *key) = (void *)BPF_FUNC_map_delete_elem; static int (*bpf_probe_read)(void *dst, int size, void *unsafe_ptr) = (void *)BPF_FUNC_probe_read; -static unsigned long long (*bpf_ktime_get_ns)(void) = + +/* Introduced in linux 5.8, see https://github.com/torvalds/linux/commit/71d19214776e61b33da48f7c1b46e522c7f78221 */ +#if LINUX_VERSION_CODE >= KERNEL_VERSION(5,8,0) +static unsigned long long (*bpf_ktime_get_boot_ns)(void) = + (void *)BPF_FUNC_ktime_get_boot_ns; +#else +/* fallback at using old, non suspend-time aware, helper */ +static unsigned long long (*bpf_ktime_get_boot_ns)(void) = (void *)BPF_FUNC_ktime_get_ns; +#endif + static int (*bpf_trace_printk)(const char *fmt, int fmt_size, ...) = (void *)BPF_FUNC_trace_printk; static void (*bpf_tail_call)(void *ctx, void *map, int index) = diff --git a/driver/bpf/plumbing_helpers.h b/driver/bpf/plumbing_helpers.h index 9a98498f8..155efd7b2 100644 --- a/driver/bpf/plumbing_helpers.h +++ b/driver/bpf/plumbing_helpers.h @@ -460,7 +460,7 @@ static __always_inline void call_filler(void *ctx, drop_flags = UF_NEVER_DROP; } - ts = settings->boot_time + bpf_ktime_get_ns(); + ts = settings->boot_time + bpf_ktime_get_boot_ns(); reset_tail_ctx(state, evt_type, ts); /* drop_event can change state->tail_ctx.evt_type */ From 89e4b20fa87c06fca2a5a358a9f5b6da107d8c12 Mon Sep 17 00:00:00 2001 From: Federico Di Pierro Date: Wed, 13 Oct 2021 10:30:25 +0200 Subject: [PATCH 096/148] update(userspace/libscap): updated libscap bpf compat header. Signed-off-by: Federico Di Pierro --- userspace/libscap/compat/bpf.h | 1642 +++++++++++++++++++++++++++----- 1 file changed, 1399 insertions(+), 243 deletions(-) diff --git a/userspace/libscap/compat/bpf.h b/userspace/libscap/compat/bpf.h index e72116afa..1ac35dbf2 100644 --- a/userspace/libscap/compat/bpf.h +++ b/userspace/libscap/compat/bpf.h @@ -9,11 +9,12 @@ #define _UAPI__LINUX_BPF_H__ #include -#include "bpf_common.h" +#include /* Extended instruction set based on top of classic BPF */ /* instruction classes */ +#define BPF_JMP32 0x06 /* jmp mode in word width */ #define BPF_ALU64 0x07 /* alu mode in double word width */ /* ld/ldx fields */ @@ -72,7 +73,7 @@ struct bpf_insn { /* Key of an a BPF_MAP_TYPE_LPM_TRIE entry */ struct bpf_lpm_trie_key { __u32 prefixlen; /* up to 32 for AF_INET, 128 for AF_INET6 */ - __u8 data[0]; /* Arbitrary size */ + __u8 data[]; /* Arbitrary size */ }; struct bpf_cgroup_storage_key { @@ -103,6 +104,15 @@ enum bpf_cmd { BPF_BTF_LOAD, BPF_BTF_GET_FD_BY_ID, BPF_TASK_FD_QUERY, + BPF_MAP_LOOKUP_AND_DELETE_ELEM, + BPF_MAP_FREEZE, + BPF_BTF_GET_NEXT_ID, + BPF_MAP_LOOKUP_BATCH, + BPF_MAP_LOOKUP_AND_DELETE_BATCH, + BPF_MAP_UPDATE_BATCH, + BPF_MAP_DELETE_BATCH, + BPF_LINK_CREATE, + BPF_LINK_UPDATE, }; enum bpf_map_type { @@ -127,8 +137,22 @@ enum bpf_map_type { BPF_MAP_TYPE_SOCKHASH, BPF_MAP_TYPE_CGROUP_STORAGE, BPF_MAP_TYPE_REUSEPORT_SOCKARRAY, + BPF_MAP_TYPE_PERCPU_CGROUP_STORAGE, + BPF_MAP_TYPE_QUEUE, + BPF_MAP_TYPE_STACK, + BPF_MAP_TYPE_SK_STORAGE, + BPF_MAP_TYPE_DEVMAP_HASH, + BPF_MAP_TYPE_STRUCT_OPS, }; +/* Note that tracing related programs such as + * BPF_PROG_TYPE_{KPROBE,TRACEPOINT,PERF_EVENT,RAW_TRACEPOINT} + * are not subject to a stable API since kernel internal data + * structures can change from release to release and may + * therefore break existing tracing BPF programs. Tracing BPF + * programs correspond to /a/ specific kernel which is to be + * analyzed, and not /a/ specific kernel /and/ all future ones. + */ enum bpf_prog_type { BPF_PROG_TYPE_UNSPEC, BPF_PROG_TYPE_SOCKET_FILTER, @@ -153,6 +177,13 @@ enum bpf_prog_type { BPF_PROG_TYPE_LIRC_MODE2, BPF_PROG_TYPE_SK_REUSEPORT, BPF_PROG_TYPE_FLOW_DISSECTOR, + BPF_PROG_TYPE_CGROUP_SYSCTL, + BPF_PROG_TYPE_RAW_TRACEPOINT_WRITABLE, + BPF_PROG_TYPE_CGROUP_SOCKOPT, + BPF_PROG_TYPE_TRACING, + BPF_PROG_TYPE_STRUCT_OPS, + BPF_PROG_TYPE_EXT, + BPF_PROG_TYPE_LSM, }; enum bpf_attach_type { @@ -174,6 +205,16 @@ enum bpf_attach_type { BPF_CGROUP_UDP6_SENDMSG, BPF_LIRC_MODE2, BPF_FLOW_DISSECTOR, + BPF_CGROUP_SYSCTL, + BPF_CGROUP_UDP4_RECVMSG, + BPF_CGROUP_UDP6_RECVMSG, + BPF_CGROUP_GETSOCKOPT, + BPF_CGROUP_SETSOCKOPT, + BPF_TRACE_RAW_TP, + BPF_TRACE_FENTRY, + BPF_TRACE_FEXIT, + BPF_MODIFY_RETURN, + BPF_LSM_MAC, __MAX_BPF_ATTACH_TYPE }; @@ -202,6 +243,11 @@ enum bpf_attach_type { * When children program makes decision (like picking TCP CA or sock bind) * parent program has a chance to override it. * + * With BPF_F_ALLOW_MULTI a new program is added to the end of the list of + * programs for a cgroup. Though it's possible to replace an old program at + * any position by also specifying BPF_F_REPLACE flag and position itself in + * replace_bpf_fd attribute. Old program at this position will be released. + * * A cgroup with MULTI or OVERRIDE flag allows any attach flags in sub-cgroups. * A cgroup with NONE doesn't allow any programs in sub-cgroups. * Ex1: @@ -220,6 +266,7 @@ enum bpf_attach_type { */ #define BPF_F_ALLOW_OVERRIDE (1U << 0) #define BPF_F_ALLOW_MULTI (1U << 1) +#define BPF_F_REPLACE (1U << 2) /* If BPF_F_STRICT_ALIGNMENT is used in BPF_PROG_LOAD command, the * verifier will perform strict alignment checking as if the kernel @@ -228,8 +275,54 @@ enum bpf_attach_type { */ #define BPF_F_STRICT_ALIGNMENT (1U << 0) -/* when bpf_ldimm64->src_reg == BPF_PSEUDO_MAP_FD, bpf_ldimm64->imm == fd */ +/* If BPF_F_ANY_ALIGNMENT is used in BPF_PROF_LOAD command, the + * verifier will allow any alignment whatsoever. On platforms + * with strict alignment requirements for loads ands stores (such + * as sparc and mips) the verifier validates that all loads and + * stores provably follow this requirement. This flag turns that + * checking and enforcement off. + * + * It is mostly used for testing when we want to validate the + * context and memory access aspects of the verifier, but because + * of an unaligned access the alignment check would trigger before + * the one we are interested in. + */ +#define BPF_F_ANY_ALIGNMENT (1U << 1) + +/* BPF_F_TEST_RND_HI32 is used in BPF_PROG_LOAD command for testing purpose. + * Verifier does sub-register def/use analysis and identifies instructions whose + * def only matters for low 32-bit, high 32-bit is never referenced later + * through implicit zero extension. Therefore verifier notifies JIT back-ends + * that it is safe to ignore clearing high 32-bit for these instructions. This + * saves some back-ends a lot of code-gen. However such optimization is not + * necessary on some arches, for example x86_64, arm64 etc, whose JIT back-ends + * hence hasn't used verifier's analysis result. But, we really want to have a + * way to be able to verify the correctness of the described optimization on + * x86_64 on which testsuites are frequently exercised. + * + * So, this flag is introduced. Once it is set, verifier will randomize high + * 32-bit for those instructions who has been identified as safe to ignore them. + * Then, if verifier is not doing correct analysis, such randomization will + * regress tests to expose bugs. + */ +#define BPF_F_TEST_RND_HI32 (1U << 2) + +/* The verifier internal test flag. Behavior is undefined */ +#define BPF_F_TEST_STATE_FREQ (1U << 3) + +/* When BPF ldimm64's insn[0].src_reg != 0 then this can have + * two extensions: + * + * insn[0].src_reg: BPF_PSEUDO_MAP_FD BPF_PSEUDO_MAP_VALUE + * insn[0].imm: map fd map fd + * insn[1].imm: 0 offset into value + * insn[0].off: 0 0 + * insn[1].off: 0 0 + * ldimm64 rewrite: address of map address of map[0]+offset + * verifier type: CONST_PTR_TO_MAP PTR_TO_MAP_VALUE + */ #define BPF_PSEUDO_MAP_FD 1 +#define BPF_PSEUDO_MAP_VALUE 2 /* when bpf_call->src_reg == BPF_PSEUDO_CALL, bpf_call->imm == pc-relative * offset to another bpf function @@ -237,33 +330,54 @@ enum bpf_attach_type { #define BPF_PSEUDO_CALL 1 /* flags for BPF_MAP_UPDATE_ELEM command */ -#define BPF_ANY 0 /* create new element or update existing */ -#define BPF_NOEXIST 1 /* create new element if it didn't exist */ -#define BPF_EXIST 2 /* update existing element */ +enum { + BPF_ANY = 0, /* create new element or update existing */ + BPF_NOEXIST = 1, /* create new element if it didn't exist */ + BPF_EXIST = 2, /* update existing element */ + BPF_F_LOCK = 4, /* spin_lock-ed map_lookup/map_update */ +}; /* flags for BPF_MAP_CREATE command */ -#define BPF_F_NO_PREALLOC (1U << 0) -/* Instead of having one common LRU list in the +enum { + BPF_F_NO_PREALLOC = (1U << 0), + /* Instead of having one common LRU list in the * BPF_MAP_TYPE_LRU_[PERCPU_]HASH map, use a percpu LRU list * which can scale and perform better. * Note, the LRU nodes (including free nodes) cannot be moved * across different LRU lists. - */ -#define BPF_F_NO_COMMON_LRU (1U << 1) -/* Specify numa node during map creation */ -#define BPF_F_NUMA_NODE (1U << 2) + */ + BPF_F_NO_COMMON_LRU = (1U << 1), + /* Specify numa node during map creation */ + BPF_F_NUMA_NODE = (1U << 2), -/* flags for BPF_PROG_QUERY */ -#define BPF_F_QUERY_EFFECTIVE (1U << 0) + /* Flags for accessing BPF object from syscall side. */ + BPF_F_RDONLY = (1U << 3), + BPF_F_WRONLY = (1U << 4), -#define BPF_OBJ_NAME_LEN 16U + /* Flag for stack_map, store build_id+offset instead of pointer */ + BPF_F_STACK_BUILD_ID = (1U << 5), + + /* Zero-initialize hash function seed. This should only be used for testing. */ + BPF_F_ZERO_SEED = (1U << 6), -/* Flags for accessing BPF object */ -#define BPF_F_RDONLY (1U << 3) -#define BPF_F_WRONLY (1U << 4) + /* Flags for accessing BPF object from program side. */ + BPF_F_RDONLY_PROG = (1U << 7), + BPF_F_WRONLY_PROG = (1U << 8), -/* Flag for stack_map, store build_id+offset instead of pointer */ -#define BPF_F_STACK_BUILD_ID (1U << 5) + /* Clone map from listener for newly accepted socket */ + BPF_F_CLONE = (1U << 9), + + /* Enable memory-mapping BPF map */ + BPF_F_MMAPABLE = (1U << 10), +}; + +/* Flags for BPF_PROG_QUERY. */ + +/* Query effective (directly attached + inherited from ancestor cgroups) + * programs that will be executed for events within a cgroup. + * attach_flags with this flag are returned only for directly attached programs. + */ +#define BPF_F_QUERY_EFFECTIVE (1U << 0) enum bpf_stack_build_id_status { /* user space need an empty entry to identify end of a trace */ @@ -284,6 +398,8 @@ struct bpf_stack_build_id { }; }; +#define BPF_OBJ_NAME_LEN 16U + union bpf_attr { struct { /* anonymous struct used by BPF_MAP_CREATE command */ __u32 map_type; /* one of enum bpf_map_type */ @@ -292,16 +408,20 @@ union bpf_attr { __u32 max_entries; /* max number of entries in a map */ __u32 map_flags; /* BPF_MAP_CREATE related * flags defined above. - */ + */ __u32 inner_map_fd; /* fd pointing to the inner map */ __u32 numa_node; /* numa node (effective only if * BPF_F_NUMA_NODE is set). - */ + */ char map_name[BPF_OBJ_NAME_LEN]; __u32 map_ifindex; /* ifindex of netdev to create on */ __u32 btf_fd; /* fd pointing to a BTF type data */ __u32 btf_key_type_id; /* BTF type_id of the key */ __u32 btf_value_type_id; /* BTF type_id of the value */ + __u32 btf_vmlinux_value_type_id;/* BTF type_id of a kernel- + * struct stored as the + * map value + */ }; struct { /* anonymous struct used by BPF_MAP_*_ELEM commands */ @@ -314,6 +434,23 @@ union bpf_attr { __u64 flags; }; + struct { /* struct used by BPF_MAP_*_BATCH commands */ + __aligned_u64 in_batch; /* start batch, + * NULL to start from beginning + */ + __aligned_u64 out_batch; /* output: next start batch */ + __aligned_u64 keys; + __aligned_u64 values; + __u32 count; /* input/output: + * input: # of key/value + * elements + * output: # of filled elements + */ + __u32 map_fd; + __u64 elem_flags; + __u64 flags; + } batch; + struct { /* anonymous struct used by BPF_PROG_LOAD command */ __u32 prog_type; /* one of enum bpf_prog_type */ __u32 insn_cnt; @@ -322,7 +459,7 @@ union bpf_attr { __u32 log_level; /* verbosity level of verifier */ __u32 log_size; /* size of user buffer */ __aligned_u64 log_buf; /* user supplied buffer */ - __u32 kern_version; /* checked when prog_type=kprobe */ + __u32 kern_version; /* not used */ __u32 prog_flags; char prog_name[BPF_OBJ_NAME_LEN]; __u32 prog_ifindex; /* ifindex of netdev to prep for */ @@ -331,6 +468,15 @@ union bpf_attr { * (context accesses, allowed helpers, etc). */ __u32 expected_attach_type; + __u32 prog_btf_fd; /* fd pointing to BTF type data */ + __u32 func_info_rec_size; /* userspace bpf_func_info size */ + __aligned_u64 func_info; /* func info */ + __u32 func_info_cnt; /* number of bpf_func_info records */ + __u32 line_info_rec_size; /* userspace bpf_line_info size */ + __aligned_u64 line_info; /* line info */ + __u32 line_info_cnt; /* number of bpf_line_info records */ + __u32 attach_btf_id; /* in-kernel BTF type id to attach to */ + __u32 attach_prog_fd; /* 0 to attach to vmlinux */ }; struct { /* anonymous struct used by BPF_OBJ_* commands */ @@ -344,17 +490,31 @@ union bpf_attr { __u32 attach_bpf_fd; /* eBPF program to attach */ __u32 attach_type; __u32 attach_flags; + __u32 replace_bpf_fd; /* previously attached eBPF + * program to replace if + * BPF_F_REPLACE is used + */ }; struct { /* anonymous struct used by BPF_PROG_TEST_RUN command */ __u32 prog_fd; __u32 retval; - __u32 data_size_in; - __u32 data_size_out; + __u32 data_size_in; /* input: len of data_in */ + __u32 data_size_out; /* input/output: len of data_out + * returns ENOSPC if data_out + * is too small. + */ __aligned_u64 data_in; __aligned_u64 data_out; __u32 repeat; __u32 duration; + __u32 ctx_size_in; /* input: len of ctx_in */ + __u32 ctx_size_out; /* input/output: len of ctx_out + * returns ENOSPC if ctx_out + * is too small. + */ + __aligned_u64 ctx_in; + __aligned_u64 ctx_out; } test; struct { /* anonymous struct used by BPF_*_GET_*_ID */ @@ -383,7 +543,7 @@ union bpf_attr { __u32 prog_cnt; } query; - struct { + struct { /* anonymous struct used by BPF_RAW_TRACEPOINT_OPEN command */ __u64 name; __u32 prog_fd; } raw_tracepoint; @@ -405,12 +565,30 @@ union bpf_attr { * tp_name for tracepoint * symbol for kprobe * filename for uprobe - */ + */ __u32 prog_id; /* output: prod_id */ __u32 fd_type; /* output: BPF_FD_TYPE_* */ __u64 probe_offset; /* output: probe_offset */ __u64 probe_addr; /* output: probe_addr */ } task_fd_query; + + struct { /* struct used by BPF_LINK_CREATE command */ + __u32 prog_fd; /* eBPF program to attach */ + __u32 target_fd; /* object to attach to */ + __u32 attach_type; /* attach type */ + __u32 flags; /* extra flags */ + } link_create; + + struct { /* struct used by BPF_LINK_UPDATE command */ + __u32 link_fd; /* link fd */ + /* new program fd to update link with */ + __u32 new_prog_fd; + __u32 flags; /* extra flags */ + /* expected link's program fd; is specified only if + * BPF_F_REPLACE flag is set in flags */ + __u32 old_prog_fd; + } link_update; + } __attribute__((aligned(8))); /* The description below is an attempt at providing documentation to eBPF @@ -461,16 +639,21 @@ union bpf_attr { * Return * 0 on success, or a negative error in case of failure. * - * int bpf_probe_read(void *dst, u32 size, const void *src) + * int bpf_probe_read(void *dst, u32 size, const void *unsafe_ptr) * Description * For tracing programs, safely attempt to read *size* bytes from - * address *src* and store the data in *dst*. + * kernel space address *unsafe_ptr* and store the data in *dst*. + * + * Generally, use bpf_probe_read_user() or bpf_probe_read_kernel() + * instead. * Return * 0 on success, or a negative error in case of failure. * * u64 bpf_ktime_get_ns(void) * Description * Return the time elapsed since system boot, in nanoseconds. + * Does not include time the system was suspended. + * See: clock_gettime(CLOCK_MONOTONIC) * Return * Current *ktime*. * @@ -484,6 +667,8 @@ union bpf_attr { * limited to five). * * Each time the helper is called, it appends a line to the trace. + * Lines are discarded while *\/sys/kernel/debug/tracing/trace* is + * open, use *\/sys/kernel/debug/tracing/trace_pipe* to avoid this. * The format of the trace is customizable, and the exact output * one will get depends on the options set in * *\/sys/kernel/debug/tracing/trace_options* (see also the @@ -561,7 +746,7 @@ union bpf_attr { * **BPF_F_INVALIDATE_HASH** (set *skb*\ **->hash**, *skb*\ * **->swhash** and *skb*\ **->l4hash** to 0). * - * A call to this helper is susceptible to change the underlaying + * A call to this helper is susceptible to change the underlying * packet buffer. Therefore, at load time, all checks on pointers * previously done by the verifier are invalidated and must be * performed again, if the helper is used in combination with @@ -586,7 +771,7 @@ union bpf_attr { * flexibility and can handle sizes larger than 2 or 4 for the * checksum to update. * - * A call to this helper is susceptible to change the underlaying + * A call to this helper is susceptible to change the underlying * packet buffer. Therefore, at load time, all checks on pointers * previously done by the verifier are invalidated and must be * performed again, if the helper is used in combination with @@ -618,7 +803,7 @@ union bpf_attr { * flexibility and can handle sizes larger than 2 or 4 for the * checksum to update. * - * A call to this helper is susceptible to change the underlaying + * A call to this helper is susceptible to change the underlying * packet buffer. Therefore, at load time, all checks on pointers * previously done by the verifier are invalidated and must be * performed again, if the helper is used in combination with @@ -673,7 +858,7 @@ union bpf_attr { * efficient, but it is handled through an action code where the * redirection happens only after the eBPF program has returned. * - * A call to this helper is susceptible to change the underlaying + * A call to this helper is susceptible to change the underlying * packet buffer. Therefore, at load time, all checks on pointers * previously done by the verifier are invalidated and must be * performed again, if the helper is used in combination with @@ -693,7 +878,7 @@ union bpf_attr { * A 64-bit integer containing the current GID and UID, and * created as such: *current_gid* **<< 32 \|** *current_uid*. * - * int bpf_get_current_comm(char *buf, u32 size_of_buf) + * int bpf_get_current_comm(void *buf, u32 size_of_buf) * Description * Copy the **comm** attribute of the current task into *buf* of * *size_of_buf*. The **comm** attribute contains the name of @@ -715,7 +900,7 @@ union bpf_attr { * based on a user-provided identifier for all traffic coming from * the tasks belonging to the related cgroup. See also the related * kernel documentation, available from the Linux sources in file - * *Documentation/cgroup-v1/net_cls.txt*. + * *Documentation/admin-guide/cgroup-v1/net_cls.rst*. * * The Linux kernel has two versions for cgroups: there are * cgroups v1 and cgroups v2. Both are available to users, who can @@ -738,7 +923,7 @@ union bpf_attr { * **ETH_P_8021Q** and **ETH_P_8021AD**, it is considered to * be **ETH_P_8021Q**. * - * A call to this helper is susceptible to change the underlaying + * A call to this helper is susceptible to change the underlying * packet buffer. Therefore, at load time, all checks on pointers * previously done by the verifier are invalidated and must be * performed again, if the helper is used in combination with @@ -750,7 +935,7 @@ union bpf_attr { * Description * Pop a VLAN header from the packet associated to *skb*. * - * A call to this helper is susceptible to change the underlaying + * A call to this helper is susceptible to change the underlying * packet buffer. Therefore, at load time, all checks on pointers * previously done by the verifier are invalidated and must be * performed again, if the helper is used in combination with @@ -889,9 +1074,9 @@ union bpf_attr { * supports redirection to the egress interface, and accepts no * flag at all. * - * The same effect can be attained with the more generic - * **bpf_redirect_map**\ (), which requires specific maps to be - * used but offers better performance. + * The same effect can also be attained with the more generic + * **bpf_redirect_map**\ (), which uses a BPF map to store the + * redirect target instead of providing it directly to the helper. * Return * For XDP, the helper returns **XDP_REDIRECT** on success or * **XDP_ABORTED** on error. For other program types, the values @@ -902,7 +1087,7 @@ union bpf_attr { * Description * Retrieve the realm or the route, that is to say the * **tclassid** field of the destination for the *skb*. The - * identifier retrieved is a user-provided tag, similar to the + * indentifier retrieved is a user-provided tag, similar to the * one used with the net_cls cgroup (see description for * **bpf_get_cgroup_classid**\ () helper), but here this tag is * held by a route (a destination entry), not by a task. @@ -922,7 +1107,7 @@ union bpf_attr { * The realm of the route for the packet associated to *skb*, or 0 * if none was found. * - * int bpf_perf_event_output(struct pt_reg *ctx, struct bpf_map *map, u64 flags, void *data, u64 size) + * int bpf_perf_event_output(void *ctx, struct bpf_map *map, u64 flags, void *data, u64 size) * Description * Write raw *data* blob into a special BPF perf event held by * *map* of type **BPF_MAP_TYPE_PERF_EVENT_ARRAY**. This perf @@ -967,7 +1152,7 @@ union bpf_attr { * Return * 0 on success, or a negative error in case of failure. * - * int bpf_skb_load_bytes(const struct sk_buff *skb, u32 offset, void *to, u32 len) + * int bpf_skb_load_bytes(const void *skb, u32 offset, void *to, u32 len) * Description * This helper was provided as an easy way to load data from a * packet. It can be used to load *len* bytes from *offset* from @@ -984,7 +1169,7 @@ union bpf_attr { * Return * 0 on success, or a negative error in case of failure. * - * int bpf_get_stackid(struct pt_reg *ctx, struct bpf_map *map, u64 flags) + * int bpf_get_stackid(void *ctx, struct bpf_map *map, u64 flags) * Description * Walk a user or a kernel stack and return its id. To achieve * this, the helper needs *ctx*, which is a pointer to the context @@ -1053,7 +1238,7 @@ union bpf_attr { * The checksum result, or a negative error code in case of * failure. * - * int bpf_skb_get_tunnel_opt(struct sk_buff *skb, u8 *opt, u32 size) + * int bpf_skb_get_tunnel_opt(struct sk_buff *skb, void *opt, u32 size) * Description * Retrieve tunnel options metadata for the packet associated to * *skb*, and store the raw tunnel option data to the buffer *opt* @@ -1071,7 +1256,7 @@ union bpf_attr { * Return * The size of the option data retrieved. * - * int bpf_skb_set_tunnel_opt(struct sk_buff *skb, u8 *opt, u32 size) + * int bpf_skb_set_tunnel_opt(struct sk_buff *skb, void *opt, u32 size) * Description * Set tunnel options metadata for the packet associated to *skb* * to the option data contained in the raw buffer *opt* of *size*. @@ -1100,7 +1285,7 @@ union bpf_attr { * All values for *flags* are reserved for future usage, and must * be left at zero. * - * A call to this helper is susceptible to change the underlaying + * A call to this helper is susceptible to change the underlying * packet buffer. Therefore, at load time, all checks on pointers * previously done by the verifier are invalidated and must be * performed again, if the helper is used in combination with @@ -1213,7 +1398,7 @@ union bpf_attr { * implicitly linearizes, unclones and drops offloads from the * *skb*. * - * A call to this helper is susceptible to change the underlaying + * A call to this helper is susceptible to change the underlying * packet buffer. Therefore, at load time, all checks on pointers * previously done by the verifier are invalidated and must be * performed again, if the helper is used in combination with @@ -1249,7 +1434,7 @@ union bpf_attr { * **bpf_skb_pull_data()** to effectively unclone the *skb* from * the very beginning in case it is indeed cloned. * - * A call to this helper is susceptible to change the underlaying + * A call to this helper is susceptible to change the underlying * packet buffer. Therefore, at load time, all checks on pointers * previously done by the verifier are invalidated and must be * performed again, if the helper is used in combination with @@ -1301,7 +1486,7 @@ union bpf_attr { * All values for *flags* are reserved for future usage, and must * be left at zero. * - * A call to this helper is susceptible to change the underlaying + * A call to this helper is susceptible to change the underlying * packet buffer. Therefore, at load time, all checks on pointers * previously done by the verifier are invalidated and must be * performed again, if the helper is used in combination with @@ -1316,7 +1501,7 @@ union bpf_attr { * can be used to prepare the packet for pushing or popping * headers. * - * A call to this helper is susceptible to change the underlaying + * A call to this helper is susceptible to change the underlying * packet buffer. Therefore, at load time, all checks on pointers * previously done by the verifier are invalidated and must be * performed again, if the helper is used in combination with @@ -1324,45 +1509,14 @@ union bpf_attr { * Return * 0 on success, or a negative error in case of failure. * - * int bpf_probe_read_str(void *dst, int size, const void *unsafe_ptr) + * int bpf_probe_read_str(void *dst, u32 size, const void *unsafe_ptr) * Description - * Copy a NUL terminated string from an unsafe address - * *unsafe_ptr* to *dst*. The *size* should include the - * terminating NUL byte. In case the string length is smaller than - * *size*, the target is not padded with further NUL bytes. If the - * string length is larger than *size*, just *size*-1 bytes are - * copied and the last byte is set to NUL. - * - * On success, the length of the copied string is returned. This - * makes this helper useful in tracing programs for reading - * strings, and more importantly to get its length at runtime. See - * the following snippet: - * - * :: - * - * SEC("kprobe/sys_open") - * void bpf_sys_open(struct pt_regs *ctx) - * { - * char buf[PATHLEN]; // PATHLEN is defined to 256 - * int res = bpf_probe_read_str(buf, sizeof(buf), - * ctx->di); - * - * // Consume buf, for example push it to - * // userspace via bpf_perf_event_output(); we - * // can use res (the string length) as event - * // size, after checking its boundaries. - * } + * Copy a NUL terminated string from an unsafe kernel address + * *unsafe_ptr* to *dst*. See bpf_probe_read_kernel_str() for + * more details. * - * In comparison, using **bpf_probe_read()** helper here instead - * to read the string would require to estimate the length at - * compile time, and would often result in copying more memory - * than necessary. - * - * Another useful use case is when parsing individual process - * arguments or individual environment variables navigating - * *current*\ **->mm->arg_start** and *current*\ - * **->mm->env_start**: using this helper and the return value, - * one can quickly iterate at the right offset of the memory area. + * Generally, use bpf_probe_read_user_str() or bpf_probe_read_kernel_str() + * instead. * Return * On success, the strictly positive length of the string, * including the trailing NUL character. On error, a negative @@ -1375,8 +1529,8 @@ union bpf_attr { * If no cookie has been set yet, generate a new cookie. Once * generated, the socket cookie remains stable for the life of the * socket. This helper can be useful for monitoring per socket - * networking traffic statistics as it provides a unique socket - * identifier per namespace. + * networking traffic statistics as it provides a global socket + * identifier that can be assumed unique. * Return * A 8-byte long non-decreasing number on success, or 0 if the * socket field is missing inside *skb*. @@ -1384,14 +1538,14 @@ union bpf_attr { * u64 bpf_get_socket_cookie(struct bpf_sock_addr *ctx) * Description * Equivalent to bpf_get_socket_cookie() helper that accepts - * *skb*, but gets socket from **struct bpf_sock_addr** contex. + * *skb*, but gets socket from **struct bpf_sock_addr** context. * Return * A 8-byte long non-decreasing number. * * u64 bpf_get_socket_cookie(struct bpf_sock_ops *ctx) * Description * Equivalent to bpf_get_socket_cookie() helper that accepts - * *skb*, but gets socket from **struct bpf_sock_ops** contex. + * *skb*, but gets socket from **struct bpf_sock_ops** context. * Return * A 8-byte long non-decreasing number. * @@ -1410,7 +1564,7 @@ union bpf_attr { * Return * 0 * - * int bpf_setsockopt(struct bpf_sock_ops *bpf_socket, int level, int optname, char *optval, int optlen) + * int bpf_setsockopt(struct bpf_sock_ops *bpf_socket, int level, int optname, void *optval, int optlen) * Description * Emulate a call to **setsockopt()** on the socket associated to * *bpf_socket*, which must be a full socket. The *level* at @@ -1432,20 +1586,38 @@ union bpf_attr { * Return * 0 on success, or a negative error in case of failure. * - * int bpf_skb_adjust_room(struct sk_buff *skb, u32 len_diff, u32 mode, u64 flags) + * int bpf_skb_adjust_room(struct sk_buff *skb, s32 len_diff, u32 mode, u64 flags) * Description * Grow or shrink the room for data in the packet associated to * *skb* by *len_diff*, and according to the selected *mode*. * - * There is a single supported mode at this time: + * There are two supported modes at this time: + * + * * **BPF_ADJ_ROOM_MAC**: Adjust room at the mac layer + * (room space is added or removed below the layer 2 header). * * * **BPF_ADJ_ROOM_NET**: Adjust room at the network layer * (room space is added or removed below the layer 3 header). * - * All values for *flags* are reserved for future usage, and must - * be left at zero. + * The following flags are supported at this time: + * + * * **BPF_F_ADJ_ROOM_FIXED_GSO**: Do not adjust gso_size. + * Adjusting mss in this way is not allowed for datagrams. * - * A call to this helper is susceptible to change the underlaying + * * **BPF_F_ADJ_ROOM_ENCAP_L3_IPV4**, + * **BPF_F_ADJ_ROOM_ENCAP_L3_IPV6**: + * Any new space is reserved to hold a tunnel header. + * Configure skb offsets and other fields accordingly. + * + * * **BPF_F_ADJ_ROOM_ENCAP_L4_GRE**, + * **BPF_F_ADJ_ROOM_ENCAP_L4_UDP**: + * Use with ENCAP_L3 flags to further specify the tunnel type. + * + * * **BPF_F_ADJ_ROOM_ENCAP_L2**\ (*len*): + * Use with ENCAP_L3/L4 flags to further specify the tunnel + * type; *len* is the length of the inner MAC header. + * + * A call to this helper is susceptible to change the underlying * packet buffer. Therefore, at load time, all checks on pointers * previously done by the verifier are invalidated and must be * performed again, if the helper is used in combination with @@ -1462,18 +1634,19 @@ union bpf_attr { * but this is only implemented for native XDP (with driver * support) as of this writing). * - * All values for *flags* are reserved for future usage, and must - * be left at zero. + * The lower two bits of *flags* are used as the return code if + * the map lookup fails. This is so that the return value can be + * one of the XDP program return codes up to XDP_TX, as chosen by + * the caller. Any higher bits in the *flags* argument must be + * unset. * - * When used to redirect packets to net devices, this helper - * provides a high performance increase over **bpf_redirect**\ (). - * This is due to various implementation details of the underlying - * mechanisms, one of which is the fact that **bpf_redirect_map**\ - * () tries to send packet as a "bulk" to the device. + * See also bpf_redirect(), which only supports redirecting to an + * ifindex, but doesn't require a map to do so. * Return - * **XDP_REDIRECT** on success, or **XDP_ABORTED** on error. + * **XDP_REDIRECT** on success, or the value of the two lower bits + * of the *flags* argument on error. * - * int bpf_sk_redirect_map(struct bpf_map *map, u32 key, u64 flags) + * int bpf_sk_redirect_map(struct sk_buff *skb, struct bpf_map *map, u32 key, u64 flags) * Description * Redirect the packet to the socket referenced by *map* (of type * **BPF_MAP_TYPE_SOCKMAP**) at index *key*. Both ingress and @@ -1524,7 +1697,7 @@ union bpf_attr { * more flexibility as the user is free to store whatever meta * data they need. * - * A call to this helper is susceptible to change the underlaying + * A call to this helper is susceptible to change the underlying * packet buffer. Therefore, at load time, all checks on pointers * previously done by the verifier are invalidated and must be * performed again, if the helper is used in combination with @@ -1593,7 +1766,7 @@ union bpf_attr { * Return * 0 on success, or a negative error in case of failure. * - * int bpf_getsockopt(struct bpf_sock_ops *bpf_socket, int level, int optname, char *optval, int optlen) + * int bpf_getsockopt(struct bpf_sock_ops *bpf_socket, int level, int optname, void *optval, int optlen) * Description * Emulate a call to **getsockopt()** on the socket associated to * *bpf_socket*, which must be a full socket. The *level* at @@ -1612,7 +1785,7 @@ union bpf_attr { * Return * 0 on success, or a negative error in case of failure. * - * int bpf_override_return(struct pt_reg *regs, u64 rc) + * int bpf_override_return(struct pt_regs *regs, u64 rc) * Description * Used for error injection, this helper uses kprobes to override * the return value of the probed function, and to set it to *rc*. @@ -1653,11 +1826,19 @@ union bpf_attr { * error if an eBPF program tries to set a callback that is not * supported in the current kernel. * - * The supported callback values that *argval* can combine are: + * *argval* is a flag array which can combine these flags: * * * **BPF_SOCK_OPS_RTO_CB_FLAG** (retransmission time out) * * **BPF_SOCK_OPS_RETRANS_CB_FLAG** (retransmission) * * **BPF_SOCK_OPS_STATE_CB_FLAG** (TCP state change) + * * **BPF_SOCK_OPS_RTT_CB_FLAG** (every RTT) + * + * Therefore, this function can be used to clear a callback flag by + * setting the appropriate bit to zero. e.g. to disable the RTO + * callback: + * + * **bpf_sock_ops_cb_flags_set(bpf_sock,** + * **bpf_sock->bpf_sock_ops_cb_flags & ~BPF_SOCK_OPS_RTO_CB_FLAG)** * * Here are some examples of where one could call such eBPF * program: @@ -1759,7 +1940,7 @@ union bpf_attr { * copied if necessary (i.e. if data was not linear and if start * and end pointers do not point to the same chunk). * - * A call to this helper is susceptible to change the underlaying + * A call to this helper is susceptible to change the underlying * packet buffer. Therefore, at load time, all checks on pointers * previously done by the verifier are invalidated and must be * performed again, if the helper is used in combination with @@ -1793,7 +1974,7 @@ union bpf_attr { * only possible to shrink the packet as of this writing, * therefore *delta* must be a negative integer. * - * A call to this helper is susceptible to change the underlaying + * A call to this helper is susceptible to change the underlying * packet buffer. Therefore, at load time, all checks on pointers * previously done by the verifier are invalidated and must be * performed again, if the helper is used in combination with @@ -1817,7 +1998,7 @@ union bpf_attr { * Return * 0 on success, or a negative error in case of failure. * - * int bpf_get_stack(struct pt_regs *regs, void *buf, u32 size, u64 flags) + * int bpf_get_stack(void *ctx, void *buf, u32 size, u64 flags) * Description * Return a user or a kernel stack in bpf program provided buffer. * To achieve this, the helper needs *ctx*, which is a pointer @@ -1850,7 +2031,7 @@ union bpf_attr { * A non-negative value equal to or less than *size* on success, * or a negative error in case of failure. * - * int bpf_skb_load_bytes_relative(const struct sk_buff *skb, u32 offset, void *to, u32 len, u32 start_header) + * int bpf_skb_load_bytes_relative(const void *skb, u32 offset, void *to, u32 len, u32 start_header) * Description * This helper is similar to **bpf_skb_load_bytes**\ () in that * it provides an easy way to load *len* bytes from *offset* @@ -1884,9 +2065,9 @@ union bpf_attr { * is set to metric from route (IPv4/IPv6 only), and ifindex * is set to the device index of the nexthop from the FIB lookup. * - * *plen* argument is the size of the passed in struct. - * *flags* argument can be a combination of one or more of the - * following values: + * *plen* argument is the size of the passed in struct. + * *flags* argument can be a combination of one or more of the + * following values: * * **BPF_FIB_LOOKUP_DIRECT** * Do a direct table lookup vs full lookup using FIB @@ -1895,15 +2076,15 @@ union bpf_attr { * Perform lookup from an egress perspective (default is * ingress). * - * *ctx* is either **struct xdp_md** for XDP programs or - * **struct sk_buff** tc cls_act programs. - * Return + * *ctx* is either **struct xdp_md** for XDP programs or + * **struct sk_buff** tc cls_act programs. + * Return * * < 0 if any input argument is invalid * * 0 on success (packet is forwarded, nexthop neighbor exists) * * > 0 one of **BPF_FIB_LKUP_RET_** codes explaining why the * packet is not forwarded or needs assist from full stack * - * int bpf_sock_hash_update(struct bpf_sock_ops_kern *skops, struct bpf_map *map, void *key, u64 flags) + * int bpf_sock_hash_update(struct bpf_sock_ops *skops, struct bpf_map *map, void *key, u64 flags) * Description * Add an entry to, or update a sockhash *map* referencing sockets. * The *skops* is used as a new value for the entry associated to @@ -1965,8 +2146,21 @@ union bpf_attr { * Only works if *skb* contains an IPv6 packet. Insert a * Segment Routing Header (**struct ipv6_sr_hdr**) inside * the IPv6 header. - * - * A call to this helper is susceptible to change the underlaying + * **BPF_LWT_ENCAP_IP** + * IP encapsulation (GRE/GUE/IPIP/etc). The outer header + * must be IPv4 or IPv6, followed by zero or more + * additional headers, up to **LWT_BPF_MAX_HEADROOM** + * total bytes in all prepended headers. Please note that + * if **skb_is_gso**\ (*skb*) is true, no more than two + * headers can be prepended, and the inner header, if + * present, should be either GRE or UDP/GUE. + * + * **BPF_LWT_ENCAP_SEG6**\ \* types can be called by BPF programs + * of type **BPF_PROG_TYPE_LWT_IN**; **BPF_LWT_ENCAP_IP** type can + * be called by bpf programs of types **BPF_PROG_TYPE_LWT_IN** and + * **BPF_PROG_TYPE_LWT_XMIT**. + * + * A call to this helper is susceptible to change the underlying * packet buffer. Therefore, at load time, all checks on pointers * previously done by the verifier are invalidated and must be * performed again, if the helper is used in combination with @@ -1981,7 +2175,7 @@ union bpf_attr { * inside the outermost IPv6 Segment Routing Header can be * modified through this helper. * - * A call to this helper is susceptible to change the underlaying + * A call to this helper is susceptible to change the underlying * packet buffer. Therefore, at load time, all checks on pointers * previously done by the verifier are invalidated and must be * performed again, if the helper is used in combination with @@ -1997,7 +2191,7 @@ union bpf_attr { * after the segments are accepted. *delta* can be as well * positive (growing) as negative (shrinking). * - * A call to this helper is susceptible to change the underlaying + * A call to this helper is susceptible to change the underlying * packet buffer. Therefore, at load time, all checks on pointers * previously done by the verifier are invalidated and must be * performed again, if the helper is used in combination with @@ -2020,13 +2214,13 @@ union bpf_attr { * Type of *param*: **int**. * **SEG6_LOCAL_ACTION_END_B6** * End.B6 action: Endpoint bound to an SRv6 policy. - * Type of param: **struct ipv6_sr_hdr**. + * Type of *param*: **struct ipv6_sr_hdr**. * **SEG6_LOCAL_ACTION_END_B6_ENCAP** * End.B6.Encap action: Endpoint bound to an SRv6 * encapsulation policy. - * Type of param: **struct ipv6_sr_hdr**. + * Type of *param*: **struct ipv6_sr_hdr**. * - * A call to this helper is susceptible to change the underlaying + * A call to this helper is susceptible to change the underlying * packet buffer. Therefore, at load time, all checks on pointers * previously done by the verifier are invalidated and must be * performed again, if the helper is used in combination with @@ -2034,52 +2228,52 @@ union bpf_attr { * Return * 0 on success, or a negative error in case of failure. * - * int bpf_rc_keydown(void *ctx, u32 protocol, u64 scancode, u32 toggle) + * int bpf_rc_repeat(void *ctx) * Description * This helper is used in programs implementing IR decoding, to - * report a successfully decoded key press with *scancode*, - * *toggle* value in the given *protocol*. The scancode will be - * translated to a keycode using the rc keymap, and reported as - * an input key down event. After a period a key up event is - * generated. This period can be extended by calling either - * **bpf_rc_keydown** () again with the same values, or calling - * **bpf_rc_repeat** (). + * report a successfully decoded repeat key message. This delays + * the generation of a key up event for previously generated + * key down event. * - * Some protocols include a toggle bit, in case the button was - * released and pressed again between consecutive scancodes. + * Some IR protocols like NEC have a special IR message for + * repeating last button, for when a button is held down. * * The *ctx* should point to the lirc sample as passed into * the program. * - * The *protocol* is the decoded protocol number (see - * **enum rc_proto** for some predefined values). - * * This helper is only available is the kernel was compiled with * the **CONFIG_BPF_LIRC_MODE2** configuration option set to * "**y**". * Return * 0 * - * int bpf_rc_repeat(void *ctx) + * int bpf_rc_keydown(void *ctx, u32 protocol, u64 scancode, u32 toggle) * Description * This helper is used in programs implementing IR decoding, to - * report a successfully decoded repeat key message. This delays - * the generation of a key up event for previously generated - * key down event. + * report a successfully decoded key press with *scancode*, + * *toggle* value in the given *protocol*. The scancode will be + * translated to a keycode using the rc keymap, and reported as + * an input key down event. After a period a key up event is + * generated. This period can be extended by calling either + * **bpf_rc_keydown**\ () again with the same values, or calling + * **bpf_rc_repeat**\ (). * - * Some IR protocols like NEC have a special IR message for - * repeating last button, for when a button is held down. + * Some protocols include a toggle bit, in case the button was + * released and pressed again between consecutive scancodes. * * The *ctx* should point to the lirc sample as passed into * the program. * + * The *protocol* is the decoded protocol number (see + * **enum rc_proto** for some predefined values). + * * This helper is only available is the kernel was compiled with * the **CONFIG_BPF_LIRC_MODE2** configuration option set to * "**y**". * Return * 0 * - * uint64_t bpf_skb_cgroup_id(struct sk_buff *skb) + * u64 bpf_skb_cgroup_id(struct sk_buff *skb) * Description * Return the cgroup v2 id of the socket associated with the *skb*. * This is roughly similar to the **bpf_get_cgroup_classid**\ () @@ -2095,6 +2289,38 @@ union bpf_attr { * Return * The id is returned or 0 in case the id could not be retrieved. * + * u64 bpf_get_current_cgroup_id(void) + * Return + * A 64-bit integer containing the current cgroup id based + * on the cgroup within which the current task is running. + * + * void *bpf_get_local_storage(void *map, u64 flags) + * Description + * Get the pointer to the local storage area. + * The type and the size of the local storage is defined + * by the *map* argument. + * The *flags* meaning is specific for each map type, + * and has to be 0 for cgroup local storage. + * + * Depending on the BPF program type, a local storage area + * can be shared between multiple instances of the BPF program, + * running simultaneously. + * + * A user should care about the synchronization by himself. + * For example, by using the **BPF_STX_XADD** instruction to alter + * the shared data. + * Return + * A pointer to the local storage area. + * + * int bpf_sk_select_reuseport(struct sk_reuseport_md *reuse, struct bpf_map *map, void *key, u64 flags) + * Description + * Select a **SO_REUSEPORT** socket from a + * **BPF_MAP_TYPE_REUSEPORT_ARRAY** *map*. + * It checks the selected socket is matching the incoming + * request in the socket buffer. + * Return + * 0 on success, or a negative error in case of failure. + * * u64 bpf_skb_ancestor_cgroup_id(struct sk_buff *skb, int ancestor_level) * Description * Return id of cgroup v2 that is ancestor of cgroup associated @@ -2113,36 +2339,702 @@ union bpf_attr { * Return * The id is returned or 0 in case the id could not be retrieved. * - * u64 bpf_get_current_cgroup_id(void) + * struct bpf_sock *bpf_sk_lookup_tcp(void *ctx, struct bpf_sock_tuple *tuple, u32 tuple_size, u64 netns, u64 flags) + * Description + * Look for TCP socket matching *tuple*, optionally in a child + * network namespace *netns*. The return value must be checked, + * and if non-**NULL**, released via **bpf_sk_release**\ (). + * + * The *ctx* should point to the context of the program, such as + * the skb or socket (depending on the hook in use). This is used + * to determine the base network namespace for the lookup. + * + * *tuple_size* must be one of: + * + * **sizeof**\ (*tuple*\ **->ipv4**) + * Look for an IPv4 socket. + * **sizeof**\ (*tuple*\ **->ipv6**) + * Look for an IPv6 socket. + * + * If the *netns* is a negative signed 32-bit integer, then the + * socket lookup table in the netns associated with the *ctx* will + * will be used. For the TC hooks, this is the netns of the device + * in the skb. For socket hooks, this is the netns of the socket. + * If *netns* is any other signed 32-bit value greater than or + * equal to zero then it specifies the ID of the netns relative to + * the netns associated with the *ctx*. *netns* values beyond the + * range of 32-bit integers are reserved for future use. + * + * All values for *flags* are reserved for future usage, and must + * be left at zero. + * + * This helper is available only if the kernel was compiled with + * **CONFIG_NET** configuration option. + * Return + * Pointer to **struct bpf_sock**, or **NULL** in case of failure. + * For sockets with reuseport option, the **struct bpf_sock** + * result is from *reuse*\ **->socks**\ [] using the hash of the + * tuple. + * + * struct bpf_sock *bpf_sk_lookup_udp(void *ctx, struct bpf_sock_tuple *tuple, u32 tuple_size, u64 netns, u64 flags) + * Description + * Look for UDP socket matching *tuple*, optionally in a child + * network namespace *netns*. The return value must be checked, + * and if non-**NULL**, released via **bpf_sk_release**\ (). + * + * The *ctx* should point to the context of the program, such as + * the skb or socket (depending on the hook in use). This is used + * to determine the base network namespace for the lookup. + * + * *tuple_size* must be one of: + * + * **sizeof**\ (*tuple*\ **->ipv4**) + * Look for an IPv4 socket. + * **sizeof**\ (*tuple*\ **->ipv6**) + * Look for an IPv6 socket. + * + * If the *netns* is a negative signed 32-bit integer, then the + * socket lookup table in the netns associated with the *ctx* will + * will be used. For the TC hooks, this is the netns of the device + * in the skb. For socket hooks, this is the netns of the socket. + * If *netns* is any other signed 32-bit value greater than or + * equal to zero then it specifies the ID of the netns relative to + * the netns associated with the *ctx*. *netns* values beyond the + * range of 32-bit integers are reserved for future use. + * + * All values for *flags* are reserved for future usage, and must + * be left at zero. + * + * This helper is available only if the kernel was compiled with + * **CONFIG_NET** configuration option. + * Return + * Pointer to **struct bpf_sock**, or **NULL** in case of failure. + * For sockets with reuseport option, the **struct bpf_sock** + * result is from *reuse*\ **->socks**\ [] using the hash of the + * tuple. + * + * int bpf_sk_release(struct bpf_sock *sock) + * Description + * Release the reference held by *sock*. *sock* must be a + * non-**NULL** pointer that was returned from + * **bpf_sk_lookup_xxx**\ (). + * Return + * 0 on success, or a negative error in case of failure. + * + * int bpf_map_push_elem(struct bpf_map *map, const void *value, u64 flags) + * Description + * Push an element *value* in *map*. *flags* is one of: + * + * **BPF_EXIST** + * If the queue/stack is full, the oldest element is + * removed to make room for this. * Return - * A 64-bit integer containing the current cgroup id based - * on the cgroup within which the current task is running. + * 0 on success, or a negative error in case of failure. * - * void* get_local_storage(void *map, u64 flags) + * int bpf_map_pop_elem(struct bpf_map *map, void *value) + * Description + * Pop an element from *map*. + * Return + * 0 on success, or a negative error in case of failure. + * + * int bpf_map_peek_elem(struct bpf_map *map, void *value) + * Description + * Get an element from *map* without removing it. + * Return + * 0 on success, or a negative error in case of failure. + * + * int bpf_msg_push_data(struct sk_msg_buff *msg, u32 start, u32 len, u64 flags) * Description - * Get the pointer to the local storage area. - * The type and the size of the local storage is defined - * by the *map* argument. - * The *flags* meaning is specific for each map type, - * and has to be 0 for cgroup local storage. + * For socket policies, insert *len* bytes into *msg* at offset + * *start*. * - * Depending on the bpf program type, a local storage area - * can be shared between multiple instances of the bpf program, - * running simultaneously. + * If a program of type **BPF_PROG_TYPE_SK_MSG** is run on a + * *msg* it may want to insert metadata or options into the *msg*. + * This can later be read and used by any of the lower layer BPF + * hooks. * - * A user should care about the synchronization by himself. - * For example, by using the BPF_STX_XADD instruction to alter - * the shared data. + * This helper may fail if under memory pressure (a malloc + * fails) in these cases BPF programs will get an appropriate + * error and BPF programs will need to handle them. * Return - * Pointer to the local storage area. + * 0 on success, or a negative error in case of failure. * - * int bpf_sk_select_reuseport(struct sk_reuseport_md *reuse, struct bpf_map *map, void *key, u64 flags) + * int bpf_msg_pop_data(struct sk_msg_buff *msg, u32 start, u32 len, u64 flags) * Description - * Select a SO_REUSEPORT sk from a BPF_MAP_TYPE_REUSEPORT_ARRAY map - * It checks the selected sk is matching the incoming - * request in the skb. + * Will remove *len* bytes from a *msg* starting at byte *start*. + * This may result in **ENOMEM** errors under certain situations if + * an allocation and copy are required due to a full ring buffer. + * However, the helper will try to avoid doing the allocation + * if possible. Other errors can occur if input parameters are + * invalid either due to *start* byte not being valid part of *msg* + * payload and/or *pop* value being to large. * Return * 0 on success, or a negative error in case of failure. + * + * int bpf_rc_pointer_rel(void *ctx, s32 rel_x, s32 rel_y) + * Description + * This helper is used in programs implementing IR decoding, to + * report a successfully decoded pointer movement. + * + * The *ctx* should point to the lirc sample as passed into + * the program. + * + * This helper is only available is the kernel was compiled with + * the **CONFIG_BPF_LIRC_MODE2** configuration option set to + * "**y**". + * Return + * 0 + * + * int bpf_spin_lock(struct bpf_spin_lock *lock) + * Description + * Acquire a spinlock represented by the pointer *lock*, which is + * stored as part of a value of a map. Taking the lock allows to + * safely update the rest of the fields in that value. The + * spinlock can (and must) later be released with a call to + * **bpf_spin_unlock**\ (\ *lock*\ ). + * + * Spinlocks in BPF programs come with a number of restrictions + * and constraints: + * + * * **bpf_spin_lock** objects are only allowed inside maps of + * types **BPF_MAP_TYPE_HASH** and **BPF_MAP_TYPE_ARRAY** (this + * list could be extended in the future). + * * BTF description of the map is mandatory. + * * The BPF program can take ONE lock at a time, since taking two + * or more could cause dead locks. + * * Only one **struct bpf_spin_lock** is allowed per map element. + * * When the lock is taken, calls (either BPF to BPF or helpers) + * are not allowed. + * * The **BPF_LD_ABS** and **BPF_LD_IND** instructions are not + * allowed inside a spinlock-ed region. + * * The BPF program MUST call **bpf_spin_unlock**\ () to release + * the lock, on all execution paths, before it returns. + * * The BPF program can access **struct bpf_spin_lock** only via + * the **bpf_spin_lock**\ () and **bpf_spin_unlock**\ () + * helpers. Loading or storing data into the **struct + * bpf_spin_lock** *lock*\ **;** field of a map is not allowed. + * * To use the **bpf_spin_lock**\ () helper, the BTF description + * of the map value must be a struct and have **struct + * bpf_spin_lock** *anyname*\ **;** field at the top level. + * Nested lock inside another struct is not allowed. + * * The **struct bpf_spin_lock** *lock* field in a map value must + * be aligned on a multiple of 4 bytes in that value. + * * Syscall with command **BPF_MAP_LOOKUP_ELEM** does not copy + * the **bpf_spin_lock** field to user space. + * * Syscall with command **BPF_MAP_UPDATE_ELEM**, or update from + * a BPF program, do not update the **bpf_spin_lock** field. + * * **bpf_spin_lock** cannot be on the stack or inside a + * networking packet (it can only be inside of a map values). + * * **bpf_spin_lock** is available to root only. + * * Tracing programs and socket filter programs cannot use + * **bpf_spin_lock**\ () due to insufficient preemption checks + * (but this may change in the future). + * * **bpf_spin_lock** is not allowed in inner maps of map-in-map. + * Return + * 0 + * + * int bpf_spin_unlock(struct bpf_spin_lock *lock) + * Description + * Release the *lock* previously locked by a call to + * **bpf_spin_lock**\ (\ *lock*\ ). + * Return + * 0 + * + * struct bpf_sock *bpf_sk_fullsock(struct bpf_sock *sk) + * Description + * This helper gets a **struct bpf_sock** pointer such + * that all the fields in this **bpf_sock** can be accessed. + * Return + * A **struct bpf_sock** pointer on success, or **NULL** in + * case of failure. + * + * struct bpf_tcp_sock *bpf_tcp_sock(struct bpf_sock *sk) + * Description + * This helper gets a **struct bpf_tcp_sock** pointer from a + * **struct bpf_sock** pointer. + * Return + * A **struct bpf_tcp_sock** pointer on success, or **NULL** in + * case of failure. + * + * int bpf_skb_ecn_set_ce(struct sk_buff *skb) + * Description + * Set ECN (Explicit Congestion Notification) field of IP header + * to **CE** (Congestion Encountered) if current value is **ECT** + * (ECN Capable Transport). Otherwise, do nothing. Works with IPv6 + * and IPv4. + * Return + * 1 if the **CE** flag is set (either by the current helper call + * or because it was already present), 0 if it is not set. + * + * struct bpf_sock *bpf_get_listener_sock(struct bpf_sock *sk) + * Description + * Return a **struct bpf_sock** pointer in **TCP_LISTEN** state. + * **bpf_sk_release**\ () is unnecessary and not allowed. + * Return + * A **struct bpf_sock** pointer on success, or **NULL** in + * case of failure. + * + * struct bpf_sock *bpf_skc_lookup_tcp(void *ctx, struct bpf_sock_tuple *tuple, u32 tuple_size, u64 netns, u64 flags) + * Description + * Look for TCP socket matching *tuple*, optionally in a child + * network namespace *netns*. The return value must be checked, + * and if non-**NULL**, released via **bpf_sk_release**\ (). + * + * This function is identical to **bpf_sk_lookup_tcp**\ (), except + * that it also returns timewait or request sockets. Use + * **bpf_sk_fullsock**\ () or **bpf_tcp_sock**\ () to access the + * full structure. + * + * This helper is available only if the kernel was compiled with + * **CONFIG_NET** configuration option. + * Return + * Pointer to **struct bpf_sock**, or **NULL** in case of failure. + * For sockets with reuseport option, the **struct bpf_sock** + * result is from *reuse*\ **->socks**\ [] using the hash of the + * tuple. + * + * int bpf_tcp_check_syncookie(struct bpf_sock *sk, void *iph, u32 iph_len, struct tcphdr *th, u32 th_len) + * Description + * Check whether *iph* and *th* contain a valid SYN cookie ACK for + * the listening socket in *sk*. + * + * *iph* points to the start of the IPv4 or IPv6 header, while + * *iph_len* contains **sizeof**\ (**struct iphdr**) or + * **sizeof**\ (**struct ip6hdr**). + * + * *th* points to the start of the TCP header, while *th_len* + * contains **sizeof**\ (**struct tcphdr**). + * + * Return + * 0 if *iph* and *th* are a valid SYN cookie ACK, or a negative + * error otherwise. + * + * int bpf_sysctl_get_name(struct bpf_sysctl *ctx, char *buf, size_t buf_len, u64 flags) + * Description + * Get name of sysctl in /proc/sys/ and copy it into provided by + * program buffer *buf* of size *buf_len*. + * + * The buffer is always NUL terminated, unless it's zero-sized. + * + * If *flags* is zero, full name (e.g. "net/ipv4/tcp_mem") is + * copied. Use **BPF_F_SYSCTL_BASE_NAME** flag to copy base name + * only (e.g. "tcp_mem"). + * Return + * Number of character copied (not including the trailing NUL). + * + * **-E2BIG** if the buffer wasn't big enough (*buf* will contain + * truncated name in this case). + * + * int bpf_sysctl_get_current_value(struct bpf_sysctl *ctx, char *buf, size_t buf_len) + * Description + * Get current value of sysctl as it is presented in /proc/sys + * (incl. newline, etc), and copy it as a string into provided + * by program buffer *buf* of size *buf_len*. + * + * The whole value is copied, no matter what file position user + * space issued e.g. sys_read at. + * + * The buffer is always NUL terminated, unless it's zero-sized. + * Return + * Number of character copied (not including the trailing NUL). + * + * **-E2BIG** if the buffer wasn't big enough (*buf* will contain + * truncated name in this case). + * + * **-EINVAL** if current value was unavailable, e.g. because + * sysctl is uninitialized and read returns -EIO for it. + * + * int bpf_sysctl_get_new_value(struct bpf_sysctl *ctx, char *buf, size_t buf_len) + * Description + * Get new value being written by user space to sysctl (before + * the actual write happens) and copy it as a string into + * provided by program buffer *buf* of size *buf_len*. + * + * User space may write new value at file position > 0. + * + * The buffer is always NUL terminated, unless it's zero-sized. + * Return + * Number of character copied (not including the trailing NUL). + * + * **-E2BIG** if the buffer wasn't big enough (*buf* will contain + * truncated name in this case). + * + * **-EINVAL** if sysctl is being read. + * + * int bpf_sysctl_set_new_value(struct bpf_sysctl *ctx, const char *buf, size_t buf_len) + * Description + * Override new value being written by user space to sysctl with + * value provided by program in buffer *buf* of size *buf_len*. + * + * *buf* should contain a string in same form as provided by user + * space on sysctl write. + * + * User space may write new value at file position > 0. To override + * the whole sysctl value file position should be set to zero. + * Return + * 0 on success. + * + * **-E2BIG** if the *buf_len* is too big. + * + * **-EINVAL** if sysctl is being read. + * + * int bpf_strtol(const char *buf, size_t buf_len, u64 flags, long *res) + * Description + * Convert the initial part of the string from buffer *buf* of + * size *buf_len* to a long integer according to the given base + * and save the result in *res*. + * + * The string may begin with an arbitrary amount of white space + * (as determined by **isspace**\ (3)) followed by a single + * optional '**-**' sign. + * + * Five least significant bits of *flags* encode base, other bits + * are currently unused. + * + * Base must be either 8, 10, 16 or 0 to detect it automatically + * similar to user space **strtol**\ (3). + * Return + * Number of characters consumed on success. Must be positive but + * no more than *buf_len*. + * + * **-EINVAL** if no valid digits were found or unsupported base + * was provided. + * + * **-ERANGE** if resulting value was out of range. + * + * int bpf_strtoul(const char *buf, size_t buf_len, u64 flags, unsigned long *res) + * Description + * Convert the initial part of the string from buffer *buf* of + * size *buf_len* to an unsigned long integer according to the + * given base and save the result in *res*. + * + * The string may begin with an arbitrary amount of white space + * (as determined by **isspace**\ (3)). + * + * Five least significant bits of *flags* encode base, other bits + * are currently unused. + * + * Base must be either 8, 10, 16 or 0 to detect it automatically + * similar to user space **strtoul**\ (3). + * Return + * Number of characters consumed on success. Must be positive but + * no more than *buf_len*. + * + * **-EINVAL** if no valid digits were found or unsupported base + * was provided. + * + * **-ERANGE** if resulting value was out of range. + * + * void *bpf_sk_storage_get(struct bpf_map *map, struct bpf_sock *sk, void *value, u64 flags) + * Description + * Get a bpf-local-storage from a *sk*. + * + * Logically, it could be thought of getting the value from + * a *map* with *sk* as the **key**. From this + * perspective, the usage is not much different from + * **bpf_map_lookup_elem**\ (*map*, **&**\ *sk*) except this + * helper enforces the key must be a full socket and the map must + * be a **BPF_MAP_TYPE_SK_STORAGE** also. + * + * Underneath, the value is stored locally at *sk* instead of + * the *map*. The *map* is used as the bpf-local-storage + * "type". The bpf-local-storage "type" (i.e. the *map*) is + * searched against all bpf-local-storages residing at *sk*. + * + * An optional *flags* (**BPF_SK_STORAGE_GET_F_CREATE**) can be + * used such that a new bpf-local-storage will be + * created if one does not exist. *value* can be used + * together with **BPF_SK_STORAGE_GET_F_CREATE** to specify + * the initial value of a bpf-local-storage. If *value* is + * **NULL**, the new bpf-local-storage will be zero initialized. + * Return + * A bpf-local-storage pointer is returned on success. + * + * **NULL** if not found or there was an error in adding + * a new bpf-local-storage. + * + * int bpf_sk_storage_delete(struct bpf_map *map, struct bpf_sock *sk) + * Description + * Delete a bpf-local-storage from a *sk*. + * Return + * 0 on success. + * + * **-ENOENT** if the bpf-local-storage cannot be found. + * + * int bpf_send_signal(u32 sig) + * Description + * Send signal *sig* to the process of the current task. + * The signal may be delivered to any of this process's threads. + * Return + * 0 on success or successfully queued. + * + * **-EBUSY** if work queue under nmi is full. + * + * **-EINVAL** if *sig* is invalid. + * + * **-EPERM** if no permission to send the *sig*. + * + * **-EAGAIN** if bpf program can try again. + * + * s64 bpf_tcp_gen_syncookie(struct bpf_sock *sk, void *iph, u32 iph_len, struct tcphdr *th, u32 th_len) + * Description + * Try to issue a SYN cookie for the packet with corresponding + * IP/TCP headers, *iph* and *th*, on the listening socket in *sk*. + * + * *iph* points to the start of the IPv4 or IPv6 header, while + * *iph_len* contains **sizeof**\ (**struct iphdr**) or + * **sizeof**\ (**struct ip6hdr**). + * + * *th* points to the start of the TCP header, while *th_len* + * contains the length of the TCP header. + * + * Return + * On success, lower 32 bits hold the generated SYN cookie in + * followed by 16 bits which hold the MSS value for that cookie, + * and the top 16 bits are unused. + * + * On failure, the returned value is one of the following: + * + * **-EINVAL** SYN cookie cannot be issued due to error + * + * **-ENOENT** SYN cookie should not be issued (no SYN flood) + * + * **-EOPNOTSUPP** kernel configuration does not enable SYN cookies + * + * **-EPROTONOSUPPORT** IP packet version is not 4 or 6 + * + * int bpf_skb_output(void *ctx, struct bpf_map *map, u64 flags, void *data, u64 size) + * Description + * Write raw *data* blob into a special BPF perf event held by + * *map* of type **BPF_MAP_TYPE_PERF_EVENT_ARRAY**. This perf + * event must have the following attributes: **PERF_SAMPLE_RAW** + * as **sample_type**, **PERF_TYPE_SOFTWARE** as **type**, and + * **PERF_COUNT_SW_BPF_OUTPUT** as **config**. + * + * The *flags* are used to indicate the index in *map* for which + * the value must be put, masked with **BPF_F_INDEX_MASK**. + * Alternatively, *flags* can be set to **BPF_F_CURRENT_CPU** + * to indicate that the index of the current CPU core should be + * used. + * + * The value to write, of *size*, is passed through eBPF stack and + * pointed by *data*. + * + * *ctx* is a pointer to in-kernel struct sk_buff. + * + * This helper is similar to **bpf_perf_event_output**\ () but + * restricted to raw_tracepoint bpf programs. + * Return + * 0 on success, or a negative error in case of failure. + * + * int bpf_probe_read_user(void *dst, u32 size, const void *unsafe_ptr) + * Description + * Safely attempt to read *size* bytes from user space address + * *unsafe_ptr* and store the data in *dst*. + * Return + * 0 on success, or a negative error in case of failure. + * + * int bpf_probe_read_kernel(void *dst, u32 size, const void *unsafe_ptr) + * Description + * Safely attempt to read *size* bytes from kernel space address + * *unsafe_ptr* and store the data in *dst*. + * Return + * 0 on success, or a negative error in case of failure. + * + * int bpf_probe_read_user_str(void *dst, u32 size, const void *unsafe_ptr) + * Description + * Copy a NUL terminated string from an unsafe user address + * *unsafe_ptr* to *dst*. The *size* should include the + * terminating NUL byte. In case the string length is smaller than + * *size*, the target is not padded with further NUL bytes. If the + * string length is larger than *size*, just *size*-1 bytes are + * copied and the last byte is set to NUL. + * + * On success, the length of the copied string is returned. This + * makes this helper useful in tracing programs for reading + * strings, and more importantly to get its length at runtime. See + * the following snippet: + * + * :: + * + * SEC("kprobe/sys_open") + * void bpf_sys_open(struct pt_regs *ctx) + * { + * char buf[PATHLEN]; // PATHLEN is defined to 256 + * int res = bpf_probe_read_user_str(buf, sizeof(buf), + * ctx->di); + * + * // Consume buf, for example push it to + * // userspace via bpf_perf_event_output(); we + * // can use res (the string length) as event + * // size, after checking its boundaries. + * } + * + * In comparison, using **bpf_probe_read_user()** helper here + * instead to read the string would require to estimate the length + * at compile time, and would often result in copying more memory + * than necessary. + * + * Another useful use case is when parsing individual process + * arguments or individual environment variables navigating + * *current*\ **->mm->arg_start** and *current*\ + * **->mm->env_start**: using this helper and the return value, + * one can quickly iterate at the right offset of the memory area. + * Return + * On success, the strictly positive length of the string, + * including the trailing NUL character. On error, a negative + * value. + * + * int bpf_probe_read_kernel_str(void *dst, u32 size, const void *unsafe_ptr) + * Description + * Copy a NUL terminated string from an unsafe kernel address *unsafe_ptr* + * to *dst*. Same semantics as with bpf_probe_read_user_str() apply. + * Return + * On success, the strictly positive length of the string, including + * the trailing NUL character. On error, a negative value. + * + * int bpf_tcp_send_ack(void *tp, u32 rcv_nxt) + * Description + * Send out a tcp-ack. *tp* is the in-kernel struct tcp_sock. + * *rcv_nxt* is the ack_seq to be sent out. + * Return + * 0 on success, or a negative error in case of failure. + * + * int bpf_send_signal_thread(u32 sig) + * Description + * Send signal *sig* to the thread corresponding to the current task. + * Return + * 0 on success or successfully queued. + * + * **-EBUSY** if work queue under nmi is full. + * + * **-EINVAL** if *sig* is invalid. + * + * **-EPERM** if no permission to send the *sig*. + * + * **-EAGAIN** if bpf program can try again. + * + * u64 bpf_jiffies64(void) + * Description + * Obtain the 64bit jiffies + * Return + * The 64 bit jiffies + * + * int bpf_read_branch_records(struct bpf_perf_event_data *ctx, void *buf, u32 size, u64 flags) + * Description + * For an eBPF program attached to a perf event, retrieve the + * branch records (struct perf_branch_entry) associated to *ctx* + * and store it in the buffer pointed by *buf* up to size + * *size* bytes. + * Return + * On success, number of bytes written to *buf*. On error, a + * negative value. + * + * The *flags* can be set to **BPF_F_GET_BRANCH_RECORDS_SIZE** to + * instead return the number of bytes required to store all the + * branch entries. If this flag is set, *buf* may be NULL. + * + * **-EINVAL** if arguments invalid or **size** not a multiple + * of sizeof(struct perf_branch_entry). + * + * **-ENOENT** if architecture does not support branch records. + * + * int bpf_get_ns_current_pid_tgid(u64 dev, u64 ino, struct bpf_pidns_info *nsdata, u32 size) + * Description + * Returns 0 on success, values for *pid* and *tgid* as seen from the current + * *namespace* will be returned in *nsdata*. + * + * On failure, the returned value is one of the following: + * + * **-EINVAL** if dev and inum supplied don't match dev_t and inode number + * with nsfs of current task, or if dev conversion to dev_t lost high bits. + * + * **-ENOENT** if pidns does not exists for the current task. + * + * int bpf_xdp_output(void *ctx, struct bpf_map *map, u64 flags, void *data, u64 size) + * Description + * Write raw *data* blob into a special BPF perf event held by + * *map* of type **BPF_MAP_TYPE_PERF_EVENT_ARRAY**. This perf + * event must have the following attributes: **PERF_SAMPLE_RAW** + * as **sample_type**, **PERF_TYPE_SOFTWARE** as **type**, and + * **PERF_COUNT_SW_BPF_OUTPUT** as **config**. + * + * The *flags* are used to indicate the index in *map* for which + * the value must be put, masked with **BPF_F_INDEX_MASK**. + * Alternatively, *flags* can be set to **BPF_F_CURRENT_CPU** + * to indicate that the index of the current CPU core should be + * used. + * + * The value to write, of *size*, is passed through eBPF stack and + * pointed by *data*. + * + * *ctx* is a pointer to in-kernel struct xdp_buff. + * + * This helper is similar to **bpf_perf_eventoutput**\ () but + * restricted to raw_tracepoint bpf programs. + * Return + * 0 on success, or a negative error in case of failure. + * + * u64 bpf_get_netns_cookie(void *ctx) + * Description + * Retrieve the cookie (generated by the kernel) of the network + * namespace the input *ctx* is associated with. The network + * namespace cookie remains stable for its lifetime and provides + * a global identifier that can be assumed unique. If *ctx* is + * NULL, then the helper returns the cookie for the initial + * network namespace. The cookie itself is very similar to that + * of bpf_get_socket_cookie() helper, but for network namespaces + * instead of sockets. + * Return + * A 8-byte long opaque number. + * + * u64 bpf_get_current_ancestor_cgroup_id(int ancestor_level) + * Description + * Return id of cgroup v2 that is ancestor of the cgroup associated + * with the current task at the *ancestor_level*. The root cgroup + * is at *ancestor_level* zero and each step down the hierarchy + * increments the level. If *ancestor_level* == level of cgroup + * associated with the current task, then return value will be the + * same as that of **bpf_get_current_cgroup_id**\ (). + * + * The helper is useful to implement policies based on cgroups + * that are upper in hierarchy than immediate cgroup associated + * with the current task. + * + * The format of returned id and helper limitations are same as in + * **bpf_get_current_cgroup_id**\ (). + * Return + * The id is returned or 0 in case the id could not be retrieved. + * + * int bpf_sk_assign(struct sk_buff *skb, struct bpf_sock *sk, u64 flags) + * Description + * Assign the *sk* to the *skb*. When combined with appropriate + * routing configuration to receive the packet towards the socket, + * will cause *skb* to be delivered to the specified socket. + * Subsequent redirection of *skb* via **bpf_redirect**\ (), + * **bpf_clone_redirect**\ () or other methods outside of BPF may + * interfere with successful delivery to the socket. + * + * This operation is only valid from TC ingress path. + * + * The *flags* argument must be zero. + * Return + * 0 on success, or a negative errno in case of failure. + * + * * **-EINVAL** Unsupported flags specified. + * * **-ENOENT** Socket is unavailable for assignment. + * * **-ENETUNREACH** Socket is unreachable (wrong netns). + * * **-EOPNOTSUPP** Unsupported operation, for example a + * call from outside of TC ingress. + * * **-ESOCKTNOSUPPORT** Socket type not supported (reuseport). + * + * u64 bpf_ktime_get_boot_ns(void) + * Description + * Return the time elapsed since system boot, in nanoseconds. + * Does include the time the system was suspended. + * See: clock_gettime(CLOCK_BOOTTIME) + * Return + * Current *ktime*. */ #define __BPF_FUNC_MAPPER(FN) \ FN(unspec), \ @@ -2228,7 +3120,49 @@ union bpf_attr { FN(get_current_cgroup_id), \ FN(get_local_storage), \ FN(sk_select_reuseport), \ - FN(skb_ancestor_cgroup_id), + FN(skb_ancestor_cgroup_id), \ + FN(sk_lookup_tcp), \ + FN(sk_lookup_udp), \ + FN(sk_release), \ + FN(map_push_elem), \ + FN(map_pop_elem), \ + FN(map_peek_elem), \ + FN(msg_push_data), \ + FN(msg_pop_data), \ + FN(rc_pointer_rel), \ + FN(spin_lock), \ + FN(spin_unlock), \ + FN(sk_fullsock), \ + FN(tcp_sock), \ + FN(skb_ecn_set_ce), \ + FN(get_listener_sock), \ + FN(skc_lookup_tcp), \ + FN(tcp_check_syncookie), \ + FN(sysctl_get_name), \ + FN(sysctl_get_current_value), \ + FN(sysctl_get_new_value), \ + FN(sysctl_set_new_value), \ + FN(strtol), \ + FN(strtoul), \ + FN(sk_storage_get), \ + FN(sk_storage_delete), \ + FN(send_signal), \ + FN(tcp_gen_syncookie), \ + FN(skb_output), \ + FN(probe_read_user), \ + FN(probe_read_kernel), \ + FN(probe_read_user_str), \ + FN(probe_read_kernel_str), \ + FN(tcp_send_ack), \ + FN(send_signal_thread), \ + FN(jiffies64), \ + FN(read_branch_records), \ + FN(get_ns_current_pid_tgid), \ + FN(xdp_output), \ + FN(get_netns_cookie), \ + FN(get_current_ancestor_cgroup_id), \ + FN(sk_assign), \ + FN(ktime_get_boot_ns), /* integer value in 'imm' field of BPF_CALL instruction selects which helper * function eBPF program intends to call @@ -2236,57 +3170,112 @@ union bpf_attr { #define __BPF_ENUM_FN(x) BPF_FUNC_ ## x enum bpf_func_id { __BPF_FUNC_MAPPER(__BPF_ENUM_FN) - __BPF_FUNC_MAX_ID, + __BPF_FUNC_MAX_ID, }; #undef __BPF_ENUM_FN /* All flags used by eBPF helper functions, placed here. */ /* BPF_FUNC_skb_store_bytes flags. */ -#define BPF_F_RECOMPUTE_CSUM (1ULL << 0) -#define BPF_F_INVALIDATE_HASH (1ULL << 1) +enum { + BPF_F_RECOMPUTE_CSUM = (1ULL << 0), + BPF_F_INVALIDATE_HASH = (1ULL << 1), +}; /* BPF_FUNC_l3_csum_replace and BPF_FUNC_l4_csum_replace flags. * First 4 bits are for passing the header field size. */ -#define BPF_F_HDR_FIELD_MASK 0xfULL +enum { + BPF_F_HDR_FIELD_MASK = 0xfULL, +}; /* BPF_FUNC_l4_csum_replace flags. */ -#define BPF_F_PSEUDO_HDR (1ULL << 4) -#define BPF_F_MARK_MANGLED_0 (1ULL << 5) -#define BPF_F_MARK_ENFORCE (1ULL << 6) +enum { + BPF_F_PSEUDO_HDR = (1ULL << 4), + BPF_F_MARK_MANGLED_0 = (1ULL << 5), + BPF_F_MARK_ENFORCE = (1ULL << 6), +}; /* BPF_FUNC_clone_redirect and BPF_FUNC_redirect flags. */ -#define BPF_F_INGRESS (1ULL << 0) +enum { + BPF_F_INGRESS = (1ULL << 0), +}; /* BPF_FUNC_skb_set_tunnel_key and BPF_FUNC_skb_get_tunnel_key flags. */ -#define BPF_F_TUNINFO_IPV6 (1ULL << 0) +enum { + BPF_F_TUNINFO_IPV6 = (1ULL << 0), +}; /* flags for both BPF_FUNC_get_stackid and BPF_FUNC_get_stack. */ -#define BPF_F_SKIP_FIELD_MASK 0xffULL -#define BPF_F_USER_STACK (1ULL << 8) -/* flags used by BPF_FUNC_get_stackid only. */ -#define BPF_F_FAST_STACK_CMP (1ULL << 9) -#define BPF_F_REUSE_STACKID (1ULL << 10) -/* flags used by BPF_FUNC_get_stack only. */ -#define BPF_F_USER_BUILD_ID (1ULL << 11) +enum { + BPF_F_SKIP_FIELD_MASK = 0xffULL, + BPF_F_USER_STACK = (1ULL << 8), + /* flags used by BPF_FUNC_get_stackid only. */ + BPF_F_FAST_STACK_CMP = (1ULL << 9), + BPF_F_REUSE_STACKID = (1ULL << 10), + /* flags used by BPF_FUNC_get_stack only. */ + BPF_F_USER_BUILD_ID = (1ULL << 11), +}; /* BPF_FUNC_skb_set_tunnel_key flags. */ -#define BPF_F_ZERO_CSUM_TX (1ULL << 1) -#define BPF_F_DONT_FRAGMENT (1ULL << 2) -#define BPF_F_SEQ_NUMBER (1ULL << 3) +enum { + BPF_F_ZERO_CSUM_TX = (1ULL << 1), + BPF_F_DONT_FRAGMENT = (1ULL << 2), + BPF_F_SEQ_NUMBER = (1ULL << 3), +}; /* BPF_FUNC_perf_event_output, BPF_FUNC_perf_event_read and * BPF_FUNC_perf_event_read_value flags. */ -#define BPF_F_INDEX_MASK 0xffffffffULL -#define BPF_F_CURRENT_CPU BPF_F_INDEX_MASK -/* BPF_FUNC_perf_event_output for sk_buff input context. */ -#define BPF_F_CTXLEN_MASK (0xfffffULL << 32) +enum { + BPF_F_INDEX_MASK = 0xffffffffULL, + BPF_F_CURRENT_CPU = BPF_F_INDEX_MASK, + /* BPF_FUNC_perf_event_output for sk_buff input context. */ + BPF_F_CTXLEN_MASK = (0xfffffULL << 32), +}; + +/* Current network namespace */ +enum { + BPF_F_CURRENT_NETNS = (-1L), +}; + +/* BPF_FUNC_skb_adjust_room flags. */ +enum { + BPF_F_ADJ_ROOM_FIXED_GSO = (1ULL << 0), + BPF_F_ADJ_ROOM_ENCAP_L3_IPV4 = (1ULL << 1), + BPF_F_ADJ_ROOM_ENCAP_L3_IPV6 = (1ULL << 2), + BPF_F_ADJ_ROOM_ENCAP_L4_GRE = (1ULL << 3), + BPF_F_ADJ_ROOM_ENCAP_L4_UDP = (1ULL << 4), +}; + +enum { + BPF_ADJ_ROOM_ENCAP_L2_MASK = 0xff, + BPF_ADJ_ROOM_ENCAP_L2_SHIFT = 56, +}; + +#define BPF_F_ADJ_ROOM_ENCAP_L2(len) (((__u64)len & \ + BPF_ADJ_ROOM_ENCAP_L2_MASK) \ + << BPF_ADJ_ROOM_ENCAP_L2_SHIFT) + +/* BPF_FUNC_sysctl_get_name flags. */ +enum { + BPF_F_SYSCTL_BASE_NAME = (1ULL << 0), +}; + +/* BPF_FUNC_sk_storage_get flags */ +enum { + BPF_SK_STORAGE_GET_F_CREATE = (1ULL << 0), +}; + +/* BPF_FUNC_read_branch_records flags. */ +enum { + BPF_F_GET_BRANCH_RECORDS_SIZE = (1ULL << 0), +}; /* Mode for BPF_FUNC_skb_adjust_room helper. */ enum bpf_adj_room_mode { BPF_ADJ_ROOM_NET, + BPF_ADJ_ROOM_MAC, }; /* Mode for BPF_FUNC_skb_load_bytes_relative helper. */ @@ -2298,9 +3287,16 @@ enum bpf_hdr_start_off { /* Encapsulation type for BPF_FUNC_lwt_push_encap helper. */ enum bpf_lwt_encap_mode { BPF_LWT_ENCAP_SEG6, - BPF_LWT_ENCAP_SEG6_INLINE + BPF_LWT_ENCAP_SEG6_INLINE, + BPF_LWT_ENCAP_IP, }; +#define __bpf_md_ptr(type, name) \ +union { \ + type name; \ + __u64 :64; \ +} __attribute__((aligned(8))) + /* user accessible mirror of in-kernel sk_buff. * new fields can only be added to the end of this structure */ @@ -2335,7 +3331,12 @@ struct __sk_buff { /* ... here. */ __u32 data_meta; - struct bpf_flow_keys *flow_keys; + __bpf_md_ptr(struct bpf_flow_keys *, flow_keys); + __u64 tstamp; + __u32 wire_len; + __u32 gso_segs; + __bpf_md_ptr(struct bpf_sock *, sk); + __u32 gso_size; }; struct bpf_tunnel_key { @@ -2369,7 +3370,7 @@ struct bpf_xfrm_state { * provide backwards compatibility with existing SCHED_CLS and SCHED_ACT * programs. * - * XDP is handled separately, see XDP_*. + * XDP is handled seprately, see XDP_*. */ enum bpf_ret_code { BPF_OK = 0, @@ -2377,7 +3378,15 @@ enum bpf_ret_code { BPF_DROP = 2, /* 3-6 reserved */ BPF_REDIRECT = 7, - /* >127 are reserved for prog type specific return codes */ + /* >127 are reserved for prog type specific return codes. + * + * BPF_LWT_REROUTE: used by BPF_PROG_TYPE_LWT_IN and + * BPF_PROG_TYPE_LWT_XMIT to indicate that skb had been + * changed and should be routed based on its new L3 header. + * (This is an L3 redirect, as opposed to L2 redirect + * represented by BPF_REDIRECT above). + */ + BPF_LWT_REROUTE = 128, }; struct bpf_sock { @@ -2387,15 +3396,80 @@ struct bpf_sock { __u32 protocol; __u32 mark; __u32 priority; - __u32 src_ip4; /* Allows 1,2,4-byte read. - * Stored in network byte order. + /* IP address also allows 1 and 2 bytes access */ + __u32 src_ip4; + __u32 src_ip6[4]; + __u32 src_port; /* host byte order */ + __u32 dst_port; /* network byte order */ + __u32 dst_ip4; + __u32 dst_ip6[4]; + __u32 state; +}; + +struct bpf_tcp_sock { + __u32 snd_cwnd; /* Sending congestion window */ + __u32 srtt_us; /* smoothed round trip time << 3 in usecs */ + __u32 rtt_min; + __u32 snd_ssthresh; /* Slow start size threshold */ + __u32 rcv_nxt; /* What we want to receive next */ + __u32 snd_nxt; /* Next sequence we send */ + __u32 snd_una; /* First byte we want an ack for */ + __u32 mss_cache; /* Cached effective mss, not including SACKS */ + __u32 ecn_flags; /* ECN status bits. */ + __u32 rate_delivered; /* saved rate sample: packets delivered */ + __u32 rate_interval_us; /* saved rate sample: time elapsed */ + __u32 packets_out; /* Packets which are "in flight" */ + __u32 retrans_out; /* Retransmitted packets out */ + __u32 total_retrans; /* Total retransmits for entire connection */ + __u32 segs_in; /* RFC4898 tcpEStatsPerfSegsIn + * total number of segments in. */ - __u32 src_ip6[4]; /* Allows 1,2,4-byte read. - * Stored in network byte order. + __u32 data_segs_in; /* RFC4898 tcpEStatsPerfDataSegsIn + * total number of data segments in. */ - __u32 src_port; /* Allows 4-byte read. - * Stored in host byte order + __u32 segs_out; /* RFC4898 tcpEStatsPerfSegsOut + * The total number of segments sent. */ + __u32 data_segs_out; /* RFC4898 tcpEStatsPerfDataSegsOut + * total number of data segments sent. + */ + __u32 lost_out; /* Lost packets */ + __u32 sacked_out; /* SACK'd packets */ + __u64 bytes_received; /* RFC4898 tcpEStatsAppHCThruOctetsReceived + * sum(delta(rcv_nxt)), or how many bytes + * were acked. + */ + __u64 bytes_acked; /* RFC4898 tcpEStatsAppHCThruOctetsAcked + * sum(delta(snd_una)), or how many bytes + * were acked. + */ + __u32 dsack_dups; /* RFC4898 tcpEStatsStackDSACKDups + * total number of DSACK blocks received + */ + __u32 delivered; /* Total data packets delivered incl. rexmits */ + __u32 delivered_ce; /* Like the above but only ECE marked packets */ + __u32 icsk_retransmits; /* Number of unrecovered [RTO] timeouts */ +}; + +struct bpf_sock_tuple { + union { + struct { + __be32 saddr; + __be32 daddr; + __be16 sport; + __be16 dport; + } ipv4; + struct { + __be32 saddr[4]; + __be32 daddr[4]; + __be16 sport; + __be16 dport; + } ipv6; + }; +}; + +struct bpf_xdp_sock { + __u32 queue_id; }; #define XDP_PACKET_HEADROOM 256 @@ -2434,8 +3508,8 @@ enum sk_action { * be added to the end of this structure */ struct sk_msg_md { - void *data; - void *data_end; + __bpf_md_ptr(void *, data); + __bpf_md_ptr(void *, data_end); __u32 family; __u32 remote_ip4; /* Stored in network byte order */ @@ -2444,6 +3518,7 @@ struct sk_msg_md { __u32 local_ip6[4]; /* Stored in network byte order */ __u32 remote_port; /* Stored in network byte order */ __u32 local_port; /* stored in host byte order */ + __u32 size; /* Total size of sk_msg */ }; struct sk_reuseport_md { @@ -2451,8 +3526,9 @@ struct sk_reuseport_md { * Start of directly accessible data. It begins from * the tcp/udp header. */ - void *data; - void *data_end; /* End of directly accessible data */ + __bpf_md_ptr(void *, data); + /* End of directly accessible data */ + __bpf_md_ptr(void *, data_end); /* * Total length of packet (starting from the tcp/udp header). * Note that the directly accessible bytes (data_end - data) @@ -2487,12 +3563,27 @@ struct bpf_prog_info { char name[BPF_OBJ_NAME_LEN]; __u32 ifindex; __u32 gpl_compatible:1; + __u32 :31; /* alignment pad */ __u64 netns_dev; __u64 netns_ino; __u32 nr_jited_ksyms; __u32 nr_jited_func_lens; __aligned_u64 jited_ksyms; __aligned_u64 jited_func_lens; + __u32 btf_id; + __u32 func_info_rec_size; + __aligned_u64 func_info; + __u32 nr_func_info; + __u32 nr_line_info; + __aligned_u64 line_info; + __aligned_u64 jited_line_info; + __u32 nr_jited_line_info; + __u32 line_info_rec_size; + __u32 jited_line_info_rec_size; + __u32 nr_prog_tags; + __aligned_u64 prog_tags; + __u64 run_time_ns; + __u64 run_cnt; } __attribute__((aligned(8))); struct bpf_map_info { @@ -2504,7 +3595,7 @@ struct bpf_map_info { __u32 map_flags; char name[BPF_OBJ_NAME_LEN]; __u32 ifindex; - __u32 :32; + __u32 btf_vmlinux_value_type_id; __u64 netns_dev; __u64 netns_ino; __u32 btf_id; @@ -2526,22 +3617,23 @@ struct bpf_sock_addr { __u32 user_family; /* Allows 4-byte read, but no write. */ __u32 user_ip4; /* Allows 1,2,4-byte read and 4-byte write. * Stored in network byte order. - */ - __u32 user_ip6[4]; /* Allows 1,2,4-byte read an 4-byte write. + */ + __u32 user_ip6[4]; /* Allows 1,2,4,8-byte read and 4,8-byte write. * Stored in network byte order. - */ + */ __u32 user_port; /* Allows 4-byte read and write. * Stored in network byte order - */ + */ __u32 family; /* Allows 4-byte read, but no write */ __u32 type; /* Allows 4-byte read, but no write */ __u32 protocol; /* Allows 4-byte read, but no write */ - __u32 msg_src_ip4; /* Allows 1,2,4-byte read an 4-byte write. + __u32 msg_src_ip4; /* Allows 1,2,4-byte read and 4-byte write. * Stored in network byte order. - */ - __u32 msg_src_ip6[4]; /* Allows 1,2,4-byte read an 4-byte write. + */ + __u32 msg_src_ip6[4]; /* Allows 1,2,4,8-byte read and 4,8-byte write. * Stored in network byte order. - */ + */ + __bpf_md_ptr(struct bpf_sock *, sk); }; /* User bpf_sock_ops struct to access socket values and specify request ops @@ -2567,7 +3659,7 @@ struct bpf_sock_ops { __u32 is_fullsock; /* Some TCP fields are only valid if * there is a full socket. If not, the * fields read as zero. - */ + */ __u32 snd_cwnd; __u32 srtt_us; /* Averaged RTT << 3 in usecs */ __u32 bpf_sock_ops_cb_flags; /* flags defined in uapi/linux/tcp.h */ @@ -2593,15 +3685,18 @@ struct bpf_sock_ops { __u32 sk_txhash; __u64 bytes_received; __u64 bytes_acked; + __bpf_md_ptr(struct bpf_sock *, sk); }; /* Definitions for bpf_sock_ops_cb_flags */ -#define BPF_SOCK_OPS_RTO_CB_FLAG (1<<0) -#define BPF_SOCK_OPS_RETRANS_CB_FLAG (1<<1) -#define BPF_SOCK_OPS_STATE_CB_FLAG (1<<2) -#define BPF_SOCK_OPS_ALL_CB_FLAGS 0x7 /* Mask of all currently - * supported cb flags - */ +enum { + BPF_SOCK_OPS_RTO_CB_FLAG = (1<<0), + BPF_SOCK_OPS_RETRANS_CB_FLAG = (1<<1), + BPF_SOCK_OPS_STATE_CB_FLAG = (1<<2), + BPF_SOCK_OPS_RTT_CB_FLAG = (1<<3), + /* Mask of all currently supported cb flags */ + BPF_SOCK_OPS_ALL_CB_FLAGS = 0xF, +}; /* List of known BPF sock_ops operators. * New entries can only be added at the end @@ -2610,50 +3705,52 @@ enum { BPF_SOCK_OPS_VOID, BPF_SOCK_OPS_TIMEOUT_INIT, /* Should return SYN-RTO value to use or * -1 if default value should be used - */ + */ BPF_SOCK_OPS_RWND_INIT, /* Should return initial advertized * window (in packets) or -1 if default * value should be used - */ + */ BPF_SOCK_OPS_TCP_CONNECT_CB, /* Calls BPF program right before an * active connection is initialized - */ + */ BPF_SOCK_OPS_ACTIVE_ESTABLISHED_CB, /* Calls BPF program when an * active connection is * established - */ + */ BPF_SOCK_OPS_PASSIVE_ESTABLISHED_CB, /* Calls BPF program when a * passive connection is * established - */ + */ BPF_SOCK_OPS_NEEDS_ECN, /* If connection's congestion control * needs ECN - */ + */ BPF_SOCK_OPS_BASE_RTT, /* Get base RTT. The correct value is * based on the path and may be * dependent on the congestion control * algorithm. In general it indicates * a congestion threshold. RTTs above * this indicate congestion - */ + */ BPF_SOCK_OPS_RTO_CB, /* Called when an RTO has triggered. * Arg1: value of icsk_retransmits * Arg2: value of icsk_rto * Arg3: whether RTO has expired - */ + */ BPF_SOCK_OPS_RETRANS_CB, /* Called when skb is retransmitted. * Arg1: sequence number of 1st byte * Arg2: # segments * Arg3: return value of * tcp_transmit_skb (0 => success) - */ + */ BPF_SOCK_OPS_STATE_CB, /* Called when TCP changes state. * Arg1: old_state * Arg2: new_state - */ + */ BPF_SOCK_OPS_TCP_LISTEN_CB, /* Called on listen(2), right after * socket transition to LISTEN state. - */ + */ + BPF_SOCK_OPS_RTT_CB, /* Called on every RTT. + */ }; /* List of TCP states. There is a build check in net/ipv4/tcp.c to detect @@ -2678,8 +3775,10 @@ enum { BPF_TCP_MAX_STATES /* Leave at the end! */ }; -#define TCP_BPF_IW 1001 /* Set TCP initial congestion window */ -#define TCP_BPF_SNDCWND_CLAMP 1002 /* Set sndcwnd_clamp */ +enum { + TCP_BPF_IW = 1001, /* Set TCP initial congestion window */ + TCP_BPF_SNDCWND_CLAMP = 1002, /* Set sndcwnd_clamp */ +}; struct bpf_perf_event_value { __u64 counter; @@ -2687,12 +3786,16 @@ struct bpf_perf_event_value { __u64 running; }; -#define BPF_DEVCG_ACC_MKNOD (1ULL << 0) -#define BPF_DEVCG_ACC_READ (1ULL << 1) -#define BPF_DEVCG_ACC_WRITE (1ULL << 2) +enum { + BPF_DEVCG_ACC_MKNOD = (1ULL << 0), + BPF_DEVCG_ACC_READ = (1ULL << 1), + BPF_DEVCG_ACC_WRITE = (1ULL << 2), +}; -#define BPF_DEVCG_DEV_BLOCK (1ULL << 0) -#define BPF_DEVCG_DEV_CHAR (1ULL << 1) +enum { + BPF_DEVCG_DEV_BLOCK = (1ULL << 0), + BPF_DEVCG_DEV_CHAR = (1ULL << 1), +}; struct bpf_cgroup_dev_ctx { /* access_type encoded as (BPF_DEVCG_ACC_* << 16) | BPF_DEVCG_DEV_* */ @@ -2708,8 +3811,10 @@ struct bpf_raw_tracepoint_args { /* DIRECT: Skip the FIB rules and go to FIB table associated with device * OUTPUT: Do lookup from egress perspective; default is ingress */ -#define BPF_FIB_LOOKUP_DIRECT BIT(0) -#define BPF_FIB_LOOKUP_OUTPUT BIT(1) +enum { + BPF_FIB_LOOKUP_DIRECT = (1U << 0), + BPF_FIB_LOOKUP_OUTPUT = (1U << 1), +}; enum { BPF_FIB_LKUP_RET_SUCCESS, /* lookup successful */ @@ -2781,6 +3886,12 @@ enum bpf_task_fd_type { BPF_FD_TYPE_URETPROBE, /* filename + offset */ }; +enum { + BPF_FLOW_DISSECTOR_F_PARSE_1ST_FRAG = (1U << 0), + BPF_FLOW_DISSECTOR_F_STOP_AT_FLOW_LABEL = (1U << 1), + BPF_FLOW_DISSECTOR_F_STOP_AT_ENCAP = (1U << 2), +}; + struct bpf_flow_keys { __u16 nhoff; __u16 thoff; @@ -2802,6 +3913,51 @@ struct bpf_flow_keys { __u32 ipv6_dst[4]; /* in6_addr; network order */ }; }; + __u32 flags; + __be32 flow_label; +}; + +struct bpf_func_info { + __u32 insn_off; + __u32 type_id; }; +#define BPF_LINE_INFO_LINE_NUM(line_col) ((line_col) >> 10) +#define BPF_LINE_INFO_LINE_COL(line_col) ((line_col) & 0x3ff) + +struct bpf_line_info { + __u32 insn_off; + __u32 file_name_off; + __u32 line_off; + __u32 line_col; +}; + +struct bpf_spin_lock { + __u32 val; +}; + +struct bpf_sysctl { + __u32 write; /* Sysctl is being read (= 0) or written (= 1). + * Allows 1,2,4-byte read, but no write. + */ + __u32 file_pos; /* Sysctl file position to read from, write to. + * Allows 1,2,4-byte read an 4-byte write. + */ +}; + +struct bpf_sockopt { + __bpf_md_ptr(struct bpf_sock *, sk); + __bpf_md_ptr(void *, optval); + __bpf_md_ptr(void *, optval_end); + + __s32 level; + __s32 optname; + __s32 optlen; + __s32 retval; +}; + +struct bpf_pidns_info { + __u32 pid; + __u32 tgid; +}; #endif /* _UAPI__LINUX_BPF_H__ */ From aaea60e41780b0f7789525767ba2595f9860becd Mon Sep 17 00:00:00 2001 From: Federico Di Pierro Date: Wed, 13 Oct 2021 10:32:44 +0200 Subject: [PATCH 097/148] feat(userspace/libscap): properly freeze eBPF const maps after creation, where BPF_MAP_FREEZE cmd is available. Signed-off-by: Federico Di Pierro --- userspace/libscap/scap_bpf.c | 21 +++++++++++++++++---- 1 file changed, 17 insertions(+), 4 deletions(-) diff --git a/userspace/libscap/scap_bpf.c b/userspace/libscap/scap_bpf.c index a7a51e91a..c88619281 100644 --- a/userspace/libscap/scap_bpf.c +++ b/userspace/libscap/scap_bpf.c @@ -140,6 +140,19 @@ static int bpf_map_create(enum bpf_map_type map_type, return sys_bpf(BPF_MAP_CREATE, &attr, sizeof(attr)); } +static int bpf_map_freeze(int fd) +{ + union bpf_attr attr; + + bzero(&attr, sizeof(attr)); + + attr.map_fd = fd; + + /* Do not check for errors as BPF_MAP_FREEZE was introduced in kernel 5.2 */ + sys_bpf(BPF_MAP_FREEZE, &attr, sizeof(attr)); + return SCAP_SUCCESS; +} + static int bpf_load_program(const struct bpf_insn *insns, enum bpf_prog_type type, size_t insns_cnt, @@ -750,7 +763,7 @@ static int32_t populate_syscall_routing_table_map(scap_t *handle) } } - return SCAP_SUCCESS; + return bpf_map_freeze(handle->m_bpf_map_fds[SYSDIG_SYSCALL_CODE_ROUTING_TABLE]); } static int32_t populate_syscall_table_map(scap_t *handle) @@ -767,7 +780,7 @@ static int32_t populate_syscall_table_map(scap_t *handle) } } - return SCAP_SUCCESS; + return bpf_map_freeze(handle->m_bpf_map_fds[SYSDIG_SYSCALL_TABLE]); } static int32_t populate_event_table_map(scap_t *handle) @@ -784,7 +797,7 @@ static int32_t populate_event_table_map(scap_t *handle) } } - return SCAP_SUCCESS; + return bpf_map_freeze(handle->m_bpf_map_fds[SYSDIG_EVENT_INFO_TABLE]); } static int32_t populate_fillers_table_map(scap_t *handle) @@ -810,7 +823,7 @@ static int32_t populate_fillers_table_map(scap_t *handle) } } - return SCAP_SUCCESS; + return bpf_map_freeze(handle->m_bpf_map_fds[SYSDIG_FILLERS_TABLE]); } // From 96e62087cd4d75ab7f2d3ce47f9dcc98c0175845 Mon Sep 17 00:00:00 2001 From: Federico Di Pierro Date: Tue, 26 Oct 2021 17:12:46 +0200 Subject: [PATCH 098/148] Restore back local "bpf_common.h" Signed-off-by: Federico Di Pierro --- userspace/libscap/compat/bpf.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/userspace/libscap/compat/bpf.h b/userspace/libscap/compat/bpf.h index 1ac35dbf2..c3fc934eb 100644 --- a/userspace/libscap/compat/bpf.h +++ b/userspace/libscap/compat/bpf.h @@ -9,7 +9,7 @@ #define _UAPI__LINUX_BPF_H__ #include -#include +#include "bpf_common.h" /* Extended instruction set based on top of classic BPF */ From 86b931d578c1452b7db38b6700cf071fcde4812b Mon Sep 17 00:00:00 2001 From: lucklypse Date: Wed, 25 Aug 2021 12:36:06 +0000 Subject: [PATCH 099/148] cleanup(libsinsp): add ipaddr and ipnet, prevent assert in debug mode Signed-off-by: lucklypse --- userspace/libsinsp/utils.cpp | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/userspace/libsinsp/utils.cpp b/userspace/libsinsp/utils.cpp index 2de13f14e..1bf61cf04 100644 --- a/userspace/libsinsp/utils.cpp +++ b/userspace/libsinsp/utils.cpp @@ -1374,6 +1374,10 @@ const char* param_type_to_string(ppm_param_type pt) return "BOOL"; case PT_IPV4ADDR: return "IPV4ADDR"; + case PT_IPADDR: + return "IPADDR"; + case PT_IPNET: + return "IPNET"; case PT_DYN: return "DYNAMIC"; case PT_FLAGS8: From ad595e5635aa9cc0955c4bf4f8c50f2c6e7e75a4 Mon Sep 17 00:00:00 2001 From: lucklypse Date: Wed, 25 Aug 2021 12:42:59 +0000 Subject: [PATCH 100/148] new(sinsp): add description field to field classes Signed-off-by: lucklypse --- userspace/libsinsp/sinsp.h | 1 + 1 file changed, 1 insertion(+) diff --git a/userspace/libsinsp/sinsp.h b/userspace/libsinsp/sinsp.h index 466bd8fe2..a9612a5cc 100644 --- a/userspace/libsinsp/sinsp.h +++ b/userspace/libsinsp/sinsp.h @@ -144,6 +144,7 @@ class filter_check_info } string m_name; ///< Field class name. + string m_desc; ///< Field class description. int32_t m_nfields; ///< Number of fields in this field group. const filtercheck_field_info* m_fields; ///< Array containing m_nfields field descriptions. uint32_t m_flags; From 5d7d9b1bc745369aa6e4d754c82bb9dfb7288579 Mon Sep 17 00:00:00 2001 From: lucklypse Date: Wed, 25 Aug 2021 12:44:20 +0000 Subject: [PATCH 101/148] new(libsinsp): add description strings for filter checks Signed-off-by: lucklypse --- userspace/libsinsp/filterchecks.cpp | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/userspace/libsinsp/filterchecks.cpp b/userspace/libsinsp/filterchecks.cpp index 6da11964c..f6c528ae8 100644 --- a/userspace/libsinsp/filterchecks.cpp +++ b/userspace/libsinsp/filterchecks.cpp @@ -143,6 +143,7 @@ sinsp_filter_check_fd::sinsp_filter_check_fd() m_fdinfo = NULL; m_info.m_name = "fd"; + m_info.m_desc = "Every syscall that has a file descriptor in its arguments has these fields set with information related to the file."; m_info.m_fields = sinsp_filter_check_fd_fields; m_info.m_nfields = sizeof(sinsp_filter_check_fd_fields) / sizeof(sinsp_filter_check_fd_fields[0]); m_info.m_flags = filter_check_info::FL_WORKS_ON_THREAD_TABLE; @@ -1862,6 +1863,7 @@ const filtercheck_field_info sinsp_filter_check_thread_fields[] = sinsp_filter_check_thread::sinsp_filter_check_thread() { m_info.m_name = "process"; + m_info.m_desc = "Additional information about the process and thread executing the syscall event."; m_info.m_fields = sinsp_filter_check_thread_fields; m_info.m_nfields = sizeof(sinsp_filter_check_thread_fields) / sizeof(sinsp_filter_check_thread_fields[0]); m_info.m_flags = filter_check_info::FL_WORKS_ON_THREAD_TABLE; @@ -2968,6 +2970,7 @@ sinsp_filter_check_event::sinsp_filter_check_event() { m_is_compare = false; m_info.m_name = "evt"; + m_info.m_desc = "Generic event fields. Note that for syscall events you can access the individual arguments/parameters of each syscall via evt.arg, e.g. evt.arg.filename."; m_info.m_fields = sinsp_filter_check_event_fields; m_info.m_nfields = sizeof(sinsp_filter_check_event_fields) / sizeof(sinsp_filter_check_event_fields[0]); m_u64val = 0; @@ -4726,6 +4729,7 @@ const filtercheck_field_info sinsp_filter_check_user_fields[] = sinsp_filter_check_user::sinsp_filter_check_user() { m_info.m_name = "user"; + m_info.m_desc = "Information about the user executing the specific event."; m_info.m_fields = sinsp_filter_check_user_fields; m_info.m_nfields = sizeof(sinsp_filter_check_user_fields) / sizeof(sinsp_filter_check_user_fields[0]); m_info.m_flags = filter_check_info::FL_WORKS_ON_THREAD_TABLE; @@ -4798,6 +4802,7 @@ const filtercheck_field_info sinsp_filter_check_group_fields[] = sinsp_filter_check_group::sinsp_filter_check_group() { m_info.m_name = "group"; + m_info.m_desc = "Information about the user group."; m_info.m_fields = sinsp_filter_check_group_fields; m_info.m_nfields = sizeof(sinsp_filter_check_group_fields) / sizeof(sinsp_filter_check_group_fields[0]); m_info.m_flags = filter_check_info::FL_WORKS_ON_THREAD_TABLE; @@ -4887,6 +4892,7 @@ sinsp_filter_check_tracer::sinsp_filter_check_tracer() { m_storage = NULL; m_info.m_name = "span"; + m_info.m_desc = "Fields used if information about distributed tracing is available."; m_info.m_fields = sinsp_filter_check_tracer_fields; m_info.m_nfields = sizeof(sinsp_filter_check_tracer_fields) / sizeof(sinsp_filter_check_tracer_fields[0]); m_converter = new sinsp_filter_check_reference(); @@ -5489,6 +5495,7 @@ sinsp_filter_check_evtin::sinsp_filter_check_evtin() { m_is_compare = false; m_info.m_name = "evtin"; + m_info.m_desc = "Fields used if information about distributed tracing is available."; m_info.m_fields = sinsp_filter_check_evtin_fields; m_info.m_nfields = sizeof(sinsp_filter_check_evtin_fields) / sizeof(sinsp_filter_check_evtin_fields[0]); m_u64val = 0; @@ -6058,6 +6065,7 @@ const filtercheck_field_info sinsp_filter_check_syslog_fields[] = sinsp_filter_check_syslog::sinsp_filter_check_syslog() { m_info.m_name = "syslog"; + m_info.m_desc = "Content of Syslog messages."; m_info.m_fields = sinsp_filter_check_syslog_fields; m_info.m_nfields = sizeof(sinsp_filter_check_syslog_fields) / sizeof(sinsp_filter_check_syslog_fields[0]); m_decoder = NULL; @@ -6138,6 +6146,7 @@ const filtercheck_field_info sinsp_filter_check_container_fields[] = sinsp_filter_check_container::sinsp_filter_check_container() { m_info.m_name = "container"; + m_info.m_desc = "Container information. If the event is not happening inside a container, both id and name will be set to 'host'."; m_info.m_fields = sinsp_filter_check_container_fields; m_info.m_nfields = sizeof(sinsp_filter_check_container_fields) / sizeof(sinsp_filter_check_container_fields[0]); m_info.m_flags = filter_check_info::FL_WORKS_ON_THREAD_TABLE; @@ -6621,6 +6630,7 @@ uint8_t* sinsp_filter_check_container::extract(sinsp_evt *evt, OUT uint32_t* len sinsp_filter_check_reference::sinsp_filter_check_reference() { m_info.m_name = ""; + m_info.m_desc = ""; m_info.m_fields = &m_finfo; m_info.m_nfields = 1; m_info.m_flags = 0; @@ -7015,6 +7025,7 @@ const filtercheck_field_info sinsp_filter_check_utils_fields[] = sinsp_filter_check_utils::sinsp_filter_check_utils() { m_info.m_name = "util"; + m_info.m_desc = ""; m_info.m_fields = sinsp_filter_check_utils_fields; m_info.m_nfields = sizeof(sinsp_filter_check_utils_fields) / sizeof(sinsp_filter_check_utils_fields[0]); m_info.m_flags = filter_check_info::FL_HIDDEN; @@ -7058,6 +7069,7 @@ const filtercheck_field_info sinsp_filter_check_fdlist_fields[] = sinsp_filter_check_fdlist::sinsp_filter_check_fdlist() { m_info.m_name = "fdlist"; + m_info.m_desc = "Poll event related fields."; m_info.m_fields = sinsp_filter_check_fdlist_fields; m_info.m_nfields = sizeof(sinsp_filter_check_fdlist_fields) / sizeof(sinsp_filter_check_fdlist_fields[0]); m_info.m_flags = filter_check_info::FL_WORKS_ON_THREAD_TABLE; @@ -7284,6 +7296,7 @@ const filtercheck_field_info sinsp_filter_check_k8s_fields[] = sinsp_filter_check_k8s::sinsp_filter_check_k8s() { m_info.m_name = "k8s"; + m_info.m_desc = "Kubernetes related context."; m_info.m_fields = sinsp_filter_check_k8s_fields; m_info.m_nfields = sizeof(sinsp_filter_check_k8s_fields) / sizeof(sinsp_filter_check_k8s_fields[0]); m_info.m_flags = filter_check_info::FL_WORKS_ON_THREAD_TABLE; @@ -7937,6 +7950,7 @@ const filtercheck_field_info sinsp_filter_check_mesos_fields[] = sinsp_filter_check_mesos::sinsp_filter_check_mesos() { m_info.m_name = "mesos"; + m_info.m_desc = "Mesos related context."; m_info.m_fields = sinsp_filter_check_mesos_fields; m_info.m_nfields = sizeof(sinsp_filter_check_mesos_fields) / sizeof(sinsp_filter_check_mesos_fields[0]); m_info.m_flags = filter_check_info::FL_WORKS_ON_THREAD_TABLE; From 4b12f65166d06cb9e36b92fb8c6158dd16e4cf00 Mon Sep 17 00:00:00 2001 From: lucklypse Date: Wed, 25 Aug 2021 12:47:37 +0000 Subject: [PATCH 102/148] change display order for filters Signed-off-by: lucklypse --- userspace/libsinsp/filter_check_list.cpp | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/userspace/libsinsp/filter_check_list.cpp b/userspace/libsinsp/filter_check_list.cpp index 0521d6cee..10dc0767d 100644 --- a/userspace/libsinsp/filter_check_list.cpp +++ b/userspace/libsinsp/filter_check_list.cpp @@ -108,14 +108,13 @@ sinsp_filter_check_list::sinsp_filter_check_list() ////////////////////////////////////////////////////////////////////////////// // ADD NEW FILTER CHECK CLASSES HERE ////////////////////////////////////////////////////////////////////////////// - add_filter_check(new sinsp_filter_check_fd()); - add_filter_check(new sinsp_filter_check_thread()); - add_filter_check(new sinsp_filter_check_gen_event()); add_filter_check(new sinsp_filter_check_event()); + add_filter_check(new sinsp_filter_check_thread()); add_filter_check(new sinsp_filter_check_user()); add_filter_check(new sinsp_filter_check_group()); - add_filter_check(new sinsp_filter_check_syslog()); add_filter_check(new sinsp_filter_check_container()); + add_filter_check(new sinsp_filter_check_fd()); + add_filter_check(new sinsp_filter_check_syslog()); add_filter_check(new sinsp_filter_check_utils()); add_filter_check(new sinsp_filter_check_fdlist()); #if !defined(CYGWING_AGENT) && !defined(MINIMAL_BUILD) @@ -124,6 +123,7 @@ sinsp_filter_check_list::sinsp_filter_check_list() #endif // !defined(CYGWING_AGENT) && !defined(MINIMAL_BUILD) add_filter_check(new sinsp_filter_check_tracer()); add_filter_check(new sinsp_filter_check_evtin()); + add_filter_check(new sinsp_filter_check_gen_event()); } sinsp_filter_check_list::~sinsp_filter_check_list() From c93acc5c16561ada82ca84223533851a2d7ced41 Mon Sep 17 00:00:00 2001 From: lucklypse Date: Wed, 25 Aug 2021 12:49:36 +0000 Subject: [PATCH 103/148] new(libsinsp): change markdown format, introduce markdown for events Signed-off-by: lucklypse --- userspace/libsinsp/fields_info.cpp | 60 +++++++++++++++++++++++------- userspace/libsinsp/fields_info.h | 2 +- 2 files changed, 48 insertions(+), 14 deletions(-) diff --git a/userspace/libsinsp/fields_info.cpp b/userspace/libsinsp/fields_info.cpp index 7e5d7840f..1ae483a1e 100644 --- a/userspace/libsinsp/fields_info.cpp +++ b/userspace/libsinsp/fields_info.cpp @@ -61,12 +61,16 @@ void list_fields(bool verbose, bool markdown, bool names_only) { if(markdown) { - printf("## Filter Class: %s\n\n", fci->m_name.c_str()); + printf("\n## Field Class: %s\n\n", fci->m_name.c_str()); + printf("%s\n\n", fci->m_desc.c_str()); + printf("Name | Type | Description\n"); + printf(":----|:-----|:-----------\n"); } else { printf("\n----------------------\n"); printf("Field Class: %s\n\n", fci->m_name.c_str()); + printf("%s\n\n", fci->m_desc.c_str()); } } @@ -85,9 +89,7 @@ void list_fields(bool verbose, bool markdown, bool names_only) } else if(markdown) { - printf("**Name**: %s \n", fld->m_name); - printf("**Description**: %s \n", fld->m_description); - printf("**Type**: %s \n\n", param_type_to_string(fld->m_type)); + printf("`%s` | %s | %s\n", fld->m_name, param_type_to_string(fld->m_type), fld->m_description); } else { @@ -140,7 +142,7 @@ void list_fields(bool verbose, bool markdown, bool names_only) } } -void list_events(sinsp* inspector) +void list_events(sinsp* inspector, bool markdown) { uint32_t j, k; string tstr; @@ -148,6 +150,12 @@ void list_events(sinsp* inspector) sinsp_evttables* einfo = inspector->get_event_info_tables(); const struct ppm_event_info* etable = einfo->m_event_info; + if(markdown) + { + printf("Falco | Dir | Event\n"); + printf(":-----|:----|:-----\n"); + } + for(j = 0; j < PPM_EVENT_MAX; j++) { const struct ppm_event_info ei = etable[j]; @@ -158,19 +166,45 @@ void list_events(sinsp* inspector) continue; } - printf("%c %s(", dir, ei.name); + if(markdown) + { + if(sinsp::simple_consumer_consider_evtnum(j)) + { + printf("Yes"); + } else { + printf("No"); + } + + printf(" | %c | **%s**(", dir, ei.name); + + for(k = 0; k < ei.nparams; k++) + { + if(k != 0) + { + printf(", "); + } + + printf("%s %s", param_type_to_string(ei.params[k].type), + ei.params[k].name); + } - for(k = 0; k < ei.nparams; k++) + printf(")\n"); + } else { - if(k != 0) + printf("%c %s(", dir, ei.name); + + for(k = 0; k < ei.nparams; k++) { - printf(", "); + if(k != 0) + { + printf(", "); + } + + printf("%s %s", param_type_to_string(ei.params[k].type), + ei.params[k].name); } - printf("%s %s", param_type_to_string(ei.params[k].type), - ei.params[k].name); + printf(")\n"); } - - printf(")\n"); } } diff --git a/userspace/libsinsp/fields_info.h b/userspace/libsinsp/fields_info.h index 93f9067e5..aeb16ac5b 100644 --- a/userspace/libsinsp/fields_info.h +++ b/userspace/libsinsp/fields_info.h @@ -25,4 +25,4 @@ class sinsp; // Printer functions // void list_fields(bool verbose, bool markdown, bool names_only=false); -void list_events(sinsp* inspector); +void list_events(sinsp* inspector, bool markdown=false); From 3923969d595fa6f0100990785b0b19179875f47d Mon Sep 17 00:00:00 2001 From: lucklypse Date: Thu, 26 Aug 2021 10:19:21 +0000 Subject: [PATCH 104/148] cleanup(libsinsp): md format cleanup Signed-off-by: lucklypse --- userspace/libsinsp/fields_info.cpp | 6 ------ userspace/libsinsp/filterchecks.cpp | 12 ++++++------ 2 files changed, 6 insertions(+), 12 deletions(-) diff --git a/userspace/libsinsp/fields_info.cpp b/userspace/libsinsp/fields_info.cpp index 1ae483a1e..5e56a6bb2 100644 --- a/userspace/libsinsp/fields_info.cpp +++ b/userspace/libsinsp/fields_info.cpp @@ -39,12 +39,6 @@ void list_fields(bool verbose, bool markdown, bool names_only) { uint32_t j, l, m; int32_t k; - - if(markdown && !names_only) - { - printf("# Filter Fields List\n\n"); - } - vector fc_plugins; sinsp::get_filtercheck_fields_info(fc_plugins); diff --git a/userspace/libsinsp/filterchecks.cpp b/userspace/libsinsp/filterchecks.cpp index f6c528ae8..40c6de635 100644 --- a/userspace/libsinsp/filterchecks.cpp +++ b/userspace/libsinsp/filterchecks.cpp @@ -2919,7 +2919,7 @@ const filtercheck_field_info sinsp_filter_check_event_fields[] = {PT_CHARBUF, EPF_NONE, PF_DIR, "evt.dir", "event direction can be either '>' for enter events or '<' for exit events."}, {PT_CHARBUF, EPF_NONE, PF_NA, "evt.type", "The name of the event (e.g. 'open')."}, {PT_UINT32, EPF_REQUIRES_ARGUMENT, PF_NA, "evt.type.is", "allows one to specify an event type, and returns 1 for events that are of that type. For example, evt.type.is.open returns 1 for open events, 0 for any other event."}, - {PT_CHARBUF, EPF_NONE, PF_NA, "syscall.type", "For system call events, the name of the system call (e.g. 'open'). Unset for other events (e.g. switch or sysdig internal events). Use this field instead of evt.type if you need to make sure that the filtered/printed value is actually a system call."}, + {PT_CHARBUF, EPF_NONE, PF_NA, "syscall.type", "For system call events, the name of the system call (e.g. 'open'). Unset for other events (e.g. switch or internal events). Use this field instead of evt.type if you need to make sure that the filtered/printed value is actually a system call."}, {PT_CHARBUF, EPF_NONE, PF_NA, "evt.category", "The event category. Example values are 'file' (for file operations like open and close), 'net' (for network operations like socket and bind), memory (for things like brk or mmap), and so on."}, {PT_INT16, EPF_NONE, PF_ID, "evt.cpu", "number of the CPU where this event happened."}, {PT_CHARBUF, EPF_NONE, PF_NA, "evt.args", "all the event arguments, aggregated into a single string."}, @@ -4883,9 +4883,9 @@ const filtercheck_field_info sinsp_filter_check_tracer_fields[] = {PT_UINT64, EPF_TABLE_ONLY, PF_DEC, "span.count", "1 for span exit events."}, {PT_UINT64, (filtercheck_field_flags) (EPF_TABLE_ONLY | EPF_REQUIRES_ARGUMENT), PF_DEC, "span.count.fortag", "1 if the span's number of tags matches the field argument, and zero for all the other ones."}, {PT_UINT64, (filtercheck_field_flags) (EPF_TABLE_ONLY | EPF_REQUIRES_ARGUMENT), PF_DEC, "span.childcount.fortag", "1 if the span's number of tags is greater than the field argument, and zero for all the other ones."}, - {PT_CHARBUF, (filtercheck_field_flags) (EPF_TABLE_ONLY | EPF_REQUIRES_ARGUMENT), PF_NA, "span.idtag", "id used by the span list csysdig view."}, - {PT_CHARBUF, EPF_TABLE_ONLY, PF_NA, "span.rawtime", "id used by the span list csysdig view."}, - {PT_CHARBUF, EPF_TABLE_ONLY, PF_NA, "span.rawparenttime", "id used by the span list csysdig view."}, + {PT_CHARBUF, (filtercheck_field_flags) (EPF_TABLE_ONLY | EPF_REQUIRES_ARGUMENT), PF_NA, "span.idtag", "id used by the span list view."}, + {PT_CHARBUF, EPF_TABLE_ONLY, PF_NA, "span.rawtime", "id used by the span list view."}, + {PT_CHARBUF, EPF_TABLE_ONLY, PF_NA, "span.rawparenttime", "id used by the span list view."}, }; sinsp_filter_check_tracer::sinsp_filter_check_tracer() @@ -6124,7 +6124,7 @@ const filtercheck_field_info sinsp_filter_check_container_fields[] = { {PT_CHARBUF, EPF_NONE, PF_NA, "container.id", "the container id."}, {PT_CHARBUF, EPF_NONE, PF_NA, "container.name", "the container name."}, - {PT_CHARBUF, EPF_NONE, PF_NA, "container.image", "the container image name (e.g. sysdig/sysdig:latest for docker, )."}, + {PT_CHARBUF, EPF_NONE, PF_NA, "container.image", "the container image name (e.g. falcosecurity/falco:latest for docker)."}, {PT_CHARBUF, EPF_NONE, PF_NA, "container.image.id", "the container image id (e.g. 6f7e2741b66b)."}, {PT_CHARBUF, EPF_NONE, PF_NA, "container.type", "the container type, eg: docker or rkt"}, {PT_BOOL, EPF_NONE, PF_NA, "container.privileged", "true for containers running as privileged, false otherwise"}, @@ -6135,7 +6135,7 @@ const filtercheck_field_info sinsp_filter_check_container_fields[] = {PT_CHARBUF, EPF_REQUIRES_ARGUMENT, PF_NA, "container.mount.mode", "the mount mode, specified by number (e.g. container.mount.mode[0]) or mount source (container.mount.mode[/usr/local]). The pathname can be a glob."}, {PT_CHARBUF, EPF_REQUIRES_ARGUMENT, PF_NA, "container.mount.rdwr", "the mount rdwr value, specified by number (e.g. container.mount.rdwr[0]) or mount source (container.mount.rdwr[/usr/local]). The pathname can be a glob."}, {PT_CHARBUF, EPF_REQUIRES_ARGUMENT, PF_NA, "container.mount.propagation", "the mount propagation value, specified by number (e.g. container.mount.propagation[0]) or mount source (container.mount.propagation[/usr/local]). The pathname can be a glob."}, - {PT_CHARBUF, EPF_NONE, PF_NA, "container.image.repository", "the container image repository (e.g. sysdig/sysdig)."}, + {PT_CHARBUF, EPF_NONE, PF_NA, "container.image.repository", "the container image repository (e.g. falcosecurity/falco)."}, {PT_CHARBUF, EPF_NONE, PF_NA, "container.image.tag", "the container image tag (e.g. stable, latest)."}, {PT_CHARBUF, EPF_NONE, PF_NA, "container.image.digest", "the container image registry digest (e.g. sha256:d977378f890d445c15e51795296e4e5062f109ce6da83e0a355fc4ad8699d27)."}, {PT_CHARBUF, EPF_NONE, PF_NA, "container.healthcheck", "The container's health check. Will be the null value (\"N/A\") if no healthcheck configured, \"NONE\" if configured but explicitly not created, and the healthcheck command line otherwise"}, From 7906f7ec416a8b67b82d92d37b25f28d545bcb8f Mon Sep 17 00:00:00 2001 From: Federico Di Pierro Date: Tue, 16 Nov 2021 09:26:54 +0100 Subject: [PATCH 105/148] fix(build): switch gtest to use main branch instead of master. Signed-off-by: Federico Di Pierro --- cmake/modules/gtest.cmake | 2 +- userspace/libsinsp/test/CMakeListsGtestInclude.cmake | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/cmake/modules/gtest.cmake b/cmake/modules/gtest.cmake index dae12e8fc..c7d030102 100644 --- a/cmake/modules/gtest.cmake +++ b/cmake/modules/gtest.cmake @@ -12,7 +12,7 @@ elseif(NOT USE_BUNDLED_GTEST) message(FATAL_ERROR "Couldn't find system gtest") endif() else() - # https://github.com/google/googletest/tree/master/googletest#incorporating-into-an-existing-cmake-project + # https://github.com/google/googletest/tree/main/googletest#incorporating-into-an-existing-cmake-project # Download and unpack googletest at configure time configure_file(CMakeListsGtestInclude.cmake ${PROJECT_BINARY_DIR}/googletest-download/CMakeLists.txt) execute_process(COMMAND ${CMAKE_COMMAND} -G "${CMAKE_GENERATOR}" . diff --git a/userspace/libsinsp/test/CMakeListsGtestInclude.cmake b/userspace/libsinsp/test/CMakeListsGtestInclude.cmake index bd0aa43cb..0e5a0b488 100644 --- a/userspace/libsinsp/test/CMakeListsGtestInclude.cmake +++ b/userspace/libsinsp/test/CMakeListsGtestInclude.cmake @@ -21,7 +21,7 @@ project(googletest-download NONE) include(ExternalProject) ExternalProject_Add(googletest GIT_REPOSITORY https://github.com/google/googletest.git - GIT_TAG master + GIT_TAG main SOURCE_DIR "${PROJECT_BINARY_DIR}/googletest-src" BINARY_DIR "${PROJECT_BINARY_DIR}/googletest-build" CONFIGURE_COMMAND "" From a358c86b1e00c433ab29096d3ecd336085db97f9 Mon Sep 17 00:00:00 2001 From: Federico Di Pierro Date: Wed, 29 Sep 2021 15:51:12 +0200 Subject: [PATCH 106/148] fix(userspace/libsinsp): split get_all_data() function in secure() and unsecure(). For SSL one, actually cycle on SSL_read() until all data is received by checking SSL_get_error(), instead of using ioctl(). Unsecure one is left untouched. Signed-off-by: Federico Di Pierro Co-authored-by: Jason Dellaluce Co-authored-by: Leonardo Grasso Co-authored-by: lucklypse --- userspace/libsinsp/socket_handler.h | 81 +++++++++++++++++++++++------ 1 file changed, 64 insertions(+), 17 deletions(-) diff --git a/userspace/libsinsp/socket_handler.h b/userspace/libsinsp/socket_handler.h index 3b7d85cff..82ee7f26b 100644 --- a/userspace/libsinsp/socket_handler.h +++ b/userspace/libsinsp/socket_handler.h @@ -353,15 +353,55 @@ class socket_data_handler } } - int get_all_data() + int get_all_data_secure() { - g_logger.log("Socket handler (" + m_id + ") Retrieving all data in blocking mode ...", - sinsp_logger::SEV_TRACE); - ssize_t rec = 0; + int rec = 0; + int processed = 0; + int counter = 0; + std::vector buf(1024, 0); + do { + rec = SSL_read(m_ssl_connection, &buf[0], buf.size()); + counter++; + if (rec > 0) + { + processed += rec; + } + int err = SSL_get_error(m_ssl_connection, rec); + switch (err) { + case SSL_ERROR_NONE: + process(&buf[0], rec, false); + break; + case SSL_ERROR_ZERO_RETURN: + throw sinsp_exception("Socket handler (" + m_id + "): Connection closed."); + break; + case SSL_ERROR_WANT_READ: + case SSL_ERROR_WANT_WRITE: + break; + default: + g_logger.log("Socket handler (" + m_id + ") received=" + std::to_string(err) + " SSL error.", sinsp_logger::SEV_TRACE); + break; + } + + // To prevent reads from entirely stalling (like in gigantic k8s environments), + // give up after reading a certain size (by default, 100MB, but configurable). + if(processed > m_data_max_b) + { + throw sinsp_exception("Socket handler (" + m_id + "): " + "read more than " + to_string(m_data_max_b / 1024 / 1024) + " MB of data from " + + m_url.to_string(false) + m_path + " (" + std::to_string(processed) + + " bytes, " + std::to_string(counter) + " reads). Giving up"); + } else { + usleep(m_data_chunk_wait_us); + } + } while (!m_msg_completed); + return processed; + } + + int get_all_data_unsecure() { + int rec = 0; std::vector buf(1024, 0); int counter = 0; - uint32_t processed = 0; - init_http_parser(); + int processed = 0; do { int count = 0; @@ -369,18 +409,11 @@ class socket_data_handler if(ioret >= 0 && count > 0) { buf.resize(count); - if(m_url.is_secure()) - { - rec = SSL_read(m_ssl_connection, &buf[0], buf.size()); - } - else - { - rec = recv(m_socket, &buf[0], buf.size(), 0); - } + rec = recv(m_socket, &buf[0], count, 0); if(rec > 0) { process(&buf[0], rec, false); - processed += (uint32_t)rec; + processed += rec; } else if(rec == 0) { @@ -394,18 +427,32 @@ class socket_data_handler // "\n\n" + data + "\n\n", sinsp_logger::SEV_TRACE); } - // To prevent reads from entirely stalling (like in gigantic k8s environments), + // To prevent reads from entirely stalling (like in gigantic k8s environments), // give up after reading a certain size (by default, 100MB, but configurable). ++counter; if(processed > m_data_max_b) { throw sinsp_exception("Socket handler (" + m_id + "): " - "read more than " + to_string(m_data_max_b / 1024 / 1024) + " MB of data from " + + "read more than " + to_string(m_data_max_b / 1024 / 1024) + " MB of data from " + m_url.to_string(false) + m_path + " (" + std::to_string(processed) + " bytes, " + std::to_string(counter) + " reads). Giving up"); } else { usleep(m_data_chunk_wait_us); } } while(!m_msg_completed); + return processed; + } + + int get_all_data() + { + g_logger.log("Socket handler (" + m_id + ") Retrieving all data in blocking mode ...", + sinsp_logger::SEV_TRACE); + int processed = 0; + init_http_parser(); + if (m_url.is_secure()) { + processed = get_all_data_secure(); + } else { + processed = get_all_data_unsecure(); + } init_http_parser(); return processed; } From 80ed029a8d960d571ea30985c674a90e1d0560da Mon Sep 17 00:00:00 2001 From: Federico Di Pierro Date: Mon, 4 Oct 2021 14:36:59 +0200 Subject: [PATCH 107/148] fix(build): bump openssl version back to 1.1.1k. Signed-off-by: Federico Di Pierro --- cmake/modules/openssl.cmake | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/cmake/modules/openssl.cmake b/cmake/modules/openssl.cmake index bc9955a99..1ed471a44 100644 --- a/cmake/modules/openssl.cmake +++ b/cmake/modules/openssl.cmake @@ -20,8 +20,8 @@ else() ExternalProject_Add(openssl PREFIX "${PROJECT_BINARY_DIR}/openssl-prefix" - URL "https://github.com/openssl/openssl/archive/OpenSSL_1_0_2u.tar.gz" - URL_HASH "SHA256=82fa58e3f273c53128c6fe7e3635ec8cda1319a10ce1ad50a987c3df0deeef05" + URL "https://github.com/openssl/openssl/archive/OpenSSL_1_1_1k.tar.gz" + URL_HASH "SHA256=b92f9d3d12043c02860e5e602e50a73ed21a69947bcc74d391f41148e9f6aa95" CONFIGURE_COMMAND ./config no-shared --prefix=${OPENSSL_INSTALL_DIR} BUILD_COMMAND ${CMD_MAKE} BUILD_IN_SOURCE 1 From 00513d0feee0db71706270aa5345c860647d0f7b Mon Sep 17 00:00:00 2001 From: Federico Di Pierro Date: Mon, 4 Oct 2021 14:51:39 +0200 Subject: [PATCH 108/148] fix(build): updated openssl to 1.1.1l. Signed-off-by: Federico Di Pierro --- cmake/modules/openssl.cmake | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/cmake/modules/openssl.cmake b/cmake/modules/openssl.cmake index 1ed471a44..6ef3de7fb 100644 --- a/cmake/modules/openssl.cmake +++ b/cmake/modules/openssl.cmake @@ -20,8 +20,8 @@ else() ExternalProject_Add(openssl PREFIX "${PROJECT_BINARY_DIR}/openssl-prefix" - URL "https://github.com/openssl/openssl/archive/OpenSSL_1_1_1k.tar.gz" - URL_HASH "SHA256=b92f9d3d12043c02860e5e602e50a73ed21a69947bcc74d391f41148e9f6aa95" + URL "https://github.com/openssl/openssl/archive/OpenSSL_1_1_1l.tar.gz" + URL_HASH "SHA256=dac036669576e83e8523afdb3971582f8b5d33993a2d6a5af87daa035f529b4f" CONFIGURE_COMMAND ./config no-shared --prefix=${OPENSSL_INSTALL_DIR} BUILD_COMMAND ${CMD_MAKE} BUILD_IN_SOURCE 1 From 4681c3865e9463abc2daedcd5f1ae7b87fd28488 Mon Sep 17 00:00:00 2001 From: Federico Di Pierro Date: Thu, 7 Oct 2021 18:06:26 +0200 Subject: [PATCH 109/148] update(userspace/libsinsp): fixed spacing. Signed-off-by: Federico Di Pierro Co-authored-by Leonardo Grasso --- userspace/libsinsp/socket_handler.h | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/userspace/libsinsp/socket_handler.h b/userspace/libsinsp/socket_handler.h index 82ee7f26b..ccbefca19 100644 --- a/userspace/libsinsp/socket_handler.h +++ b/userspace/libsinsp/socket_handler.h @@ -387,7 +387,7 @@ class socket_data_handler if(processed > m_data_max_b) { throw sinsp_exception("Socket handler (" + m_id + "): " - "read more than " + to_string(m_data_max_b / 1024 / 1024) + " MB of data from " + + "read more than " + to_string(m_data_max_b / 1024 / 1024) + " MB of data from " + m_url.to_string(false) + m_path + " (" + std::to_string(processed) + " bytes, " + std::to_string(counter) + " reads). Giving up"); } else { @@ -1693,4 +1693,4 @@ template const std::string socket_data_handler::HTTP_VERSION_11 = "1.1"; #endif // HAS_CAPTURE -#endif // MINIMAL_BUILD \ No newline at end of file +#endif // MINIMAL_BUILD From fe4bb4a32e20b308f817ba596058104588067000 Mon Sep 17 00:00:00 2001 From: Federico Di Pierro Date: Thu, 7 Oct 2021 18:08:17 +0200 Subject: [PATCH 110/148] update(userspace/libsinsp): fixed spacing. Signed-off-by: Federico Di Pierro Co-authored-by: Leonardo Grasso --- userspace/libsinsp/socket_handler.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/userspace/libsinsp/socket_handler.h b/userspace/libsinsp/socket_handler.h index ccbefca19..efac31015 100644 --- a/userspace/libsinsp/socket_handler.h +++ b/userspace/libsinsp/socket_handler.h @@ -433,7 +433,7 @@ class socket_data_handler if(processed > m_data_max_b) { throw sinsp_exception("Socket handler (" + m_id + "): " - "read more than " + to_string(m_data_max_b / 1024 / 1024) + " MB of data from " + + "read more than " + to_string(m_data_max_b / 1024 / 1024) + " MB of data from " + m_url.to_string(false) + m_path + " (" + std::to_string(processed) + " bytes, " + std::to_string(counter) + " reads). Giving up"); } From 824747a4895cd9e5ed95c44b0572d2e370f5ddab Mon Sep 17 00:00:00 2001 From: Federico Di Pierro Date: Thu, 11 Nov 2021 15:31:29 +0100 Subject: [PATCH 111/148] update(build): define OPENSSL_LIBRARIES even when built with bundled deps. Signed-off-by: Federico Di Pierro --- cmake/modules/openssl.cmake | 1 + userspace/libsinsp/CMakeLists.txt | 20 +++++++------------- 2 files changed, 8 insertions(+), 13 deletions(-) diff --git a/cmake/modules/openssl.cmake b/cmake/modules/openssl.cmake index 6ef3de7fb..b2c9ce238 100644 --- a/cmake/modules/openssl.cmake +++ b/cmake/modules/openssl.cmake @@ -14,6 +14,7 @@ else() set(OPENSSL_INCLUDE_DIR "${PROJECT_BINARY_DIR}/openssl-prefix/src/openssl/include") set(OPENSSL_LIBRARY_SSL "${OPENSSL_INSTALL_DIR}/lib/libssl.a") set(OPENSSL_LIBRARY_CRYPTO "${OPENSSL_INSTALL_DIR}/lib/libcrypto.a") + set(OPENSSL_LIBRARIES "${OPENSSL_LIBRARY_SSL} ${OPENSSL_LIBRARY_CRYPTO}") if(NOT TARGET openssl) message(STATUS "Using bundled openssl in '${OPENSSL_BUNDLE_DIR}'") diff --git a/userspace/libsinsp/CMakeLists.txt b/userspace/libsinsp/CMakeLists.txt index f14c54c63..b6f2dad2e 100644 --- a/userspace/libsinsp/CMakeLists.txt +++ b/userspace/libsinsp/CMakeLists.txt @@ -249,23 +249,17 @@ if(NOT WIN32) endif() endif() # NOT APPLE - if(USE_BUNDLED_OPENSSL AND NOT MINIMAL_BUILD) - list(APPEND SINSP_LIBRARIES - "${OPENSSL_LIBRARY_SSL}" - "${OPENSSL_LIBRARY_CRYPTO}") - else() - list(APPEND SINSP_LIBRARIES - "${OPENSSL_LIBRARIES}") - endif() + list(APPEND SINSP_LIBRARIES + "${OPENSSL_LIBRARIES}") - if(WITH_CHISEL) + if(WITH_CHISEL) list(APPEND SINSP_LIBRARIES "${LUAJIT_LIB}") - endif() + endif() - list(APPEND SINSP_LIBRARIES - dl - pthread) + list(APPEND SINSP_LIBRARIES + dl + pthread) else() if(WITH_CHISEL) list(APPEND SINSP_LIBRARIES From c48853e3797081817ba0de6ae25b52a8ece9ee69 Mon Sep 17 00:00:00 2001 From: Federico Di Pierro Date: Thu, 11 Nov 2021 16:19:23 +0100 Subject: [PATCH 112/148] fix(build): avoid quotation marks. Signed-off-by: Federico Di Pierro --- cmake/modules/openssl.cmake | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/cmake/modules/openssl.cmake b/cmake/modules/openssl.cmake index b2c9ce238..7926ffbe9 100644 --- a/cmake/modules/openssl.cmake +++ b/cmake/modules/openssl.cmake @@ -14,7 +14,7 @@ else() set(OPENSSL_INCLUDE_DIR "${PROJECT_BINARY_DIR}/openssl-prefix/src/openssl/include") set(OPENSSL_LIBRARY_SSL "${OPENSSL_INSTALL_DIR}/lib/libssl.a") set(OPENSSL_LIBRARY_CRYPTO "${OPENSSL_INSTALL_DIR}/lib/libcrypto.a") - set(OPENSSL_LIBRARIES "${OPENSSL_LIBRARY_SSL} ${OPENSSL_LIBRARY_CRYPTO}") + set(OPENSSL_LIBRARIES ${OPENSSL_LIBRARY_SSL} ${OPENSSL_LIBRARY_CRYPTO}) if(NOT TARGET openssl) message(STATUS "Using bundled openssl in '${OPENSSL_BUNDLE_DIR}'") From c5a8d93162e0b1803226187a2c1adddd859877f5 Mon Sep 17 00:00:00 2001 From: Federico Di Pierro Date: Thu, 11 Nov 2021 16:19:59 +0100 Subject: [PATCH 113/148] update(userspace/libsinsp): refactored get_all_data(). Signed-off-by: Federico Di Pierro --- userspace/libsinsp/socket_handler.h | 130 ++++++++++++---------------- 1 file changed, 55 insertions(+), 75 deletions(-) diff --git a/userspace/libsinsp/socket_handler.h b/userspace/libsinsp/socket_handler.h index efac31015..e21dc2017 100644 --- a/userspace/libsinsp/socket_handler.h +++ b/userspace/libsinsp/socket_handler.h @@ -353,106 +353,86 @@ class socket_data_handler } } - int get_all_data_secure() + int get_all_data_secure(std::vector &buf) { - int rec = 0; int processed = 0; - int counter = 0; - std::vector buf(1024, 0); - do { - rec = SSL_read(m_ssl_connection, &buf[0], buf.size()); - counter++; - if (rec > 0) - { - processed += rec; - } - int err = SSL_get_error(m_ssl_connection, rec); - switch (err) { - case SSL_ERROR_NONE: - process(&buf[0], rec, false); - break; - case SSL_ERROR_ZERO_RETURN: + int rec = SSL_read(m_ssl_connection, &buf[0], buf.size()); + if (rec > 0) + { + processed += rec; + } + int err = SSL_get_error(m_ssl_connection, rec); + switch (err) { + case SSL_ERROR_NONE: + process(&buf[0], rec, false); + break; + case SSL_ERROR_ZERO_RETURN: + throw sinsp_exception("SSL Socket handler (" + m_id + "): Connection closed."); + break; + case SSL_ERROR_WANT_READ: + case SSL_ERROR_WANT_WRITE: + break; + default: + g_logger.log("SSL Socket handler (" + m_id + ") received=" + std::to_string(err) + " SSL error.", sinsp_logger::SEV_TRACE); + break; + } + return processed; + } + + int get_all_data_unsecure(std::vector &buf) { + int rec; + int processed = 0; + int count = 0; + int ioret = ioctl(m_socket, FIONREAD, &count); + if(ioret >= 0 && count > 0) + { + buf.resize(count); + rec = recv(m_socket, &buf[0], count, 0); + switch (rec) { + case 0: throw sinsp_exception("Socket handler (" + m_id + "): Connection closed."); break; - case SSL_ERROR_WANT_READ: - case SSL_ERROR_WANT_WRITE: + case -1: + throw sinsp_exception("Socket handler (" + m_id + "): " + strerror(errno)); break; default: - g_logger.log("Socket handler (" + m_id + ") received=" + std::to_string(err) + " SSL error.", sinsp_logger::SEV_TRACE); + process(&buf[0], rec, false); + processed = rec; break; } - - // To prevent reads from entirely stalling (like in gigantic k8s environments), - // give up after reading a certain size (by default, 100MB, but configurable). - if(processed > m_data_max_b) - { - throw sinsp_exception("Socket handler (" + m_id + "): " - "read more than " + to_string(m_data_max_b / 1024 / 1024) + " MB of data from " + - m_url.to_string(false) + m_path + " (" + std::to_string(processed) + - " bytes, " + std::to_string(counter) + " reads). Giving up"); - } else { - usleep(m_data_chunk_wait_us); - } - } while (!m_msg_completed); + } return processed; } - int get_all_data_unsecure() { - int rec = 0; - std::vector buf(1024, 0); - int counter = 0; + int get_all_data() + { int processed = 0; + int counter = 0; + std::vector buf(1024, 0); + + g_logger.log("Socket handler (" + m_id + ") Retrieving all data in blocking mode ...", + sinsp_logger::SEV_TRACE); + init_http_parser(); do { - int count = 0; - int ioret = ioctl(m_socket, FIONREAD, &count); - if(ioret >= 0 && count > 0) - { - buf.resize(count); - rec = recv(m_socket, &buf[0], count, 0); - if(rec > 0) - { - process(&buf[0], rec, false); - processed += rec; - } - else if(rec == 0) - { - throw sinsp_exception("Socket handler (" + m_id + "): Connection closed."); - } - else if(rec < 0) - { - throw sinsp_exception("Socket handler (" + m_id + "): " + strerror(errno)); - } - //g_logger.log("Socket handler (" + m_id + ") received=" + std::to_string(rec) + - // "\n\n" + data + "\n\n", sinsp_logger::SEV_TRACE); + if (m_url.is_secure()) { + processed += get_all_data_secure(buf); + } else { + processed += get_all_data_unsecure(buf); } - // To prevent reads from entirely stalling (like in gigantic k8s environments), // give up after reading a certain size (by default, 100MB, but configurable). ++counter; if(processed > m_data_max_b) { throw sinsp_exception("Socket handler (" + m_id + "): " - "read more than " + to_string(m_data_max_b / 1024 / 1024) + " MB of data from " + + "read more than " + to_string(m_data_max_b / 1024 / 1024) + " MB of data from " + m_url.to_string(false) + m_path + " (" + std::to_string(processed) + " bytes, " + std::to_string(counter) + " reads). Giving up"); + } else { + usleep(m_data_chunk_wait_us); } - else { usleep(m_data_chunk_wait_us); } } while(!m_msg_completed); - return processed; - } - - int get_all_data() - { - g_logger.log("Socket handler (" + m_id + ") Retrieving all data in blocking mode ...", - sinsp_logger::SEV_TRACE); - int processed = 0; - init_http_parser(); - if (m_url.is_secure()) { - processed = get_all_data_secure(); - } else { - processed = get_all_data_unsecure(); - } init_http_parser(); return processed; } From e586e82795208fd8b988865d1d946ebefbc8c25b Mon Sep 17 00:00:00 2001 From: Federico Di Pierro Date: Mon, 15 Nov 2021 16:39:15 +0100 Subject: [PATCH 114/148] update(build): updated luajit to v2.1.0-beta3. Signed-off-by: Federico Di Pierro --- cmake/modules/luajit.cmake | 10 ++++------ userspace/chisel/chisel.cpp | 6 +++--- userspace/chisel/lua_parser.cpp | 2 +- userspace/chisel/lua_parser_api.cpp | 2 +- 4 files changed, 9 insertions(+), 11 deletions(-) diff --git a/cmake/modules/luajit.cmake b/cmake/modules/luajit.cmake index 2fee5f5f4..037f602ad 100644 --- a/cmake/modules/luajit.cmake +++ b/cmake/modules/luajit.cmake @@ -38,7 +38,6 @@ else() PREFIX "${PROJECT_BINARY_DIR}/luajit-prefix" GIT_REPOSITORY "https://github.com/moonjit/moonjit" GIT_TAG "2.1.2" - PATCH_COMMAND sed -i "s/luaL_reg/luaL_Reg/g" ${PROJECT_SOURCE_DIR}/userspace/libsinsp/chisel.cpp && sed -i "s/luaL_reg/luaL_Reg/g" ${PROJECT_SOURCE_DIR}/userspace/libsinsp/lua_parser.cpp && sed -i "s/luaL_getn/lua_objlen /g" ${PROJECT_SOURCE_DIR}/userspace/libsinsp/lua_parser_api.cpp CONFIGURE_COMMAND "" BUILD_COMMAND ${CMD_MAKE} BUILD_IN_SOURCE 1 @@ -49,7 +48,6 @@ else() PREFIX "${PROJECT_BINARY_DIR}/luajit-prefix" GIT_REPOSITORY "https://github.com/linux-on-ibm-z/LuaJIT.git" GIT_TAG "v2.1" - PATCH_COMMAND sed -i "s/luaL_reg/luaL_Reg/g" ${PROJECT_SOURCE_DIR}/userspace/libsinsp/chisel.cpp && sed -i "s/luaL_reg/luaL_Reg/g" ${PROJECT_SOURCE_DIR}/userspace/libsinsp/lua_parser.cpp && sed -i "s/luaL_getn/lua_objlen /g" ${PROJECT_SOURCE_DIR}/userspace/libsinsp/lua_parser_api.cpp CONFIGURE_COMMAND "" BUILD_COMMAND ${CMD_MAKE} BUILD_IN_SOURCE 1 @@ -58,8 +56,8 @@ else() else() ExternalProject_Add(luajit PREFIX "${PROJECT_BINARY_DIR}/luajit-prefix" - URL "https://github.com/LuaJIT/LuaJIT/archive/v2.0.3.tar.gz" - URL_HASH "SHA256=8da3d984495a11ba1bce9a833ba60e18b532ca0641e7d90d97fafe85ff014baa" + URL "https://github.com/LuaJIT/LuaJIT/archive/v2.1.0-beta3.tar.gz" + URL_HASH "SHA256=409f7fe570d3c16558e594421c47bdd130238323c9d6fd6c83dedd2aaeb082a8" CONFIGURE_COMMAND "" BUILD_COMMAND ${CMD_MAKE} BUILD_IN_SOURCE 1 @@ -69,8 +67,8 @@ else() else() ExternalProject_Add(luajit PREFIX "${PROJECT_BINARY_DIR}/luajit-prefix" - URL "https://github.com/LuaJIT/LuaJIT/archive/v2.0.3.tar.gz" - URL_HASH "SHA256=8da3d984495a11ba1bce9a833ba60e18b532ca0641e7d90d97fafe85ff014baa" + URL "https://github.com/LuaJIT/LuaJIT/archive/v2.1.0-beta3.tar.gz" + URL_HASH "SHA256=409f7fe570d3c16558e594421c47bdd130238323c9d6fd6c83dedd2aaeb082a8" CONFIGURE_COMMAND "" BUILD_COMMAND msvcbuild.bat BUILD_BYPRODUCTS ${LUAJIT_LIB} diff --git a/userspace/chisel/chisel.cpp b/userspace/chisel/chisel.cpp index cd5a96792..4743619b5 100644 --- a/userspace/chisel/chisel.cpp +++ b/userspace/chisel/chisel.cpp @@ -95,7 +95,7 @@ void lua_stackdump(lua_State *L) // Lua callbacks /////////////////////////////////////////////////////////////////////////////// #ifdef HAS_LUA_CHISELS -const static struct luaL_reg ll_sysdig [] = +const static struct luaL_Reg ll_sysdig [] = { {"set_filter", &lua_cbacks::set_global_filter}, {"set_snaplen", &lua_cbacks::set_snaplen}, @@ -131,7 +131,7 @@ const static struct luaL_reg ll_sysdig [] = {NULL,NULL} }; -const static struct luaL_reg ll_chisel [] = +const static struct luaL_Reg ll_chisel [] = { {"request_field", &lua_cbacks::request_field}, {"set_filter", &lua_cbacks::set_filter}, @@ -143,7 +143,7 @@ const static struct luaL_reg ll_chisel [] = {NULL,NULL} }; -const static struct luaL_reg ll_evt [] = +const static struct luaL_Reg ll_evt [] = { {"field", &lua_cbacks::field}, {"get_num", &lua_cbacks::get_num}, diff --git a/userspace/chisel/lua_parser.cpp b/userspace/chisel/lua_parser.cpp index f42d47694..76b37afad 100644 --- a/userspace/chisel/lua_parser.cpp +++ b/userspace/chisel/lua_parser.cpp @@ -29,7 +29,7 @@ extern "C" { #include "lauxlib.h" } -const static struct luaL_reg ll_filter [] = +const static struct luaL_Reg ll_filter [] = { {"rel_expr", &lua_parser_cbacks::rel_expr}, {"bool_op", &lua_parser_cbacks::bool_op}, diff --git a/userspace/chisel/lua_parser_api.cpp b/userspace/chisel/lua_parser_api.cpp index 5a7e13cb5..34bc8a2b1 100644 --- a/userspace/chisel/lua_parser_api.cpp +++ b/userspace/chisel/lua_parser_api.cpp @@ -266,7 +266,7 @@ int lua_parser_cbacks::rel_expr(lua_State *ls) string err = "Got non-table as in-expression operand for field " + string(fld); throw sinsp_exception(err); } - int n = luaL_getn(ls, 4); /* get size of table */ + int n = lua_objlen(ls, 4); /* get size of table */ for (i=1; i<=n; i++) { lua_rawgeti(ls, 4, i); From 4f320f32ef36738a282e0e801bb114a749f8a00c Mon Sep 17 00:00:00 2001 From: Federico Di Pierro Date: Tue, 16 Nov 2021 14:37:51 +0100 Subject: [PATCH 115/148] update(build): updated luaJIT to most recent version. Signed-off-by: Federico Di Pierro --- cmake/modules/luajit.cmake | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/cmake/modules/luajit.cmake b/cmake/modules/luajit.cmake index 037f602ad..6cbaefc51 100644 --- a/cmake/modules/luajit.cmake +++ b/cmake/modules/luajit.cmake @@ -56,8 +56,8 @@ else() else() ExternalProject_Add(luajit PREFIX "${PROJECT_BINARY_DIR}/luajit-prefix" - URL "https://github.com/LuaJIT/LuaJIT/archive/v2.1.0-beta3.tar.gz" - URL_HASH "SHA256=409f7fe570d3c16558e594421c47bdd130238323c9d6fd6c83dedd2aaeb082a8" + GIT_REPOSITORY "https://github.com/LuaJIT/LuaJIT" + GIT_TAG "f3c856915b4ce7ccd24341e8ac73e8a9fd934171" CONFIGURE_COMMAND "" BUILD_COMMAND ${CMD_MAKE} BUILD_IN_SOURCE 1 From 2c4f5f45c9e1e3074440e32b25fbec295a55686a Mon Sep 17 00:00:00 2001 From: Leonardo Grasso Date: Tue, 11 May 2021 09:59:00 +0200 Subject: [PATCH 116/148] chore(userspace/libscap): applying patch from Falco This patch was originally used in Falco to adapt probe naming. Reference: https://github.com/falcosecurity/falco/blob/1653898f4f8f64f94c0729415190349c8305a40b/cmake/modules/falcosecurity-libs-repo/patch/libscap.patch Co-authored-by: Lorenzo Fontana Co-Authored-By: Leonardo Di Donato Signed-off-by: Leonardo Grasso --- userspace/libscap/scap.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/userspace/libscap/scap.c b/userspace/libscap/scap.c index 434ac8e4d..d3d89ac14 100644 --- a/userspace/libscap/scap.c +++ b/userspace/libscap/scap.c @@ -56,7 +56,7 @@ limitations under the License. //#define NDEBUG #include -static const char *SYSDIG_BPF_PROBE_ENV = "SYSDIG_BPF_PROBE"; +static const char *SYSDIG_BPF_PROBE_ENV = "FALCO_BPF_PROBE"; // // Probe version string size @@ -146,7 +146,7 @@ static uint32_t get_max_consumers() { #ifndef _WIN32 uint32_t max; - FILE *pfile = fopen("/sys/module/" PROBE_DEVICE_NAME "_probe/parameters/max_consumers", "r"); + FILE *pfile = fopen("/sys/module/" PROBE_DEVICE_NAME "/parameters/max_consumers", "r"); if(pfile != NULL) { int w = fscanf(pfile, "%"PRIu32, &max); @@ -220,7 +220,7 @@ scap_t* scap_open_live_int(char *error, int32_t *rc, return NULL; } - snprintf(buf, sizeof(buf), "%s/.sysdig/%s-bpf.o", home, PROBE_NAME); + snprintf(buf, sizeof(buf), "%s/.falco/%s-bpf.o", home, PROBE_NAME); bpf_probe = buf; } } @@ -380,7 +380,7 @@ scap_t* scap_open_live_int(char *error, int32_t *rc, else if(errno == EBUSY) { uint32_t curr_max_consumers = get_max_consumers(); - snprintf(error, SCAP_LASTERR_SIZE, "Too many sysdig instances attached to device %s. Current value for /sys/module/" PROBE_DEVICE_NAME "_probe/parameters/max_consumers is '%"PRIu32"'.", filename, curr_max_consumers); + snprintf(error, SCAP_LASTERR_SIZE, "Too many Falco instances attached to device %s. Current value for /sys/module/" PROBE_DEVICE_NAME "/parameters/max_consumers is '%"PRIu32"'.", filename, curr_max_consumers); } else { @@ -2509,7 +2509,7 @@ int32_t scap_disable_dynamic_snaplen(scap_t* handle) const char* scap_get_host_root() { - char* p = getenv("SYSDIG_HOST_ROOT"); + char* p = getenv("HOST_ROOT"); static char env_str[SCAP_MAX_PATH_SIZE + 1]; static bool inited = false; if (! inited) { From 0b30d1eb8e3047371223802ae41343ea22d041a2 Mon Sep 17 00:00:00 2001 From: Leonardo Grasso Date: Tue, 11 May 2021 11:25:08 +0200 Subject: [PATCH 117/148] update(userspace/libscap): make probe naming more configurable, set defaults to "scap" Signed-off-by: Leonardo Grasso --- userspace/libscap/CMakeLists.txt | 18 ++++++++++++++++-- userspace/libscap/scap.c | 16 +++++++--------- 2 files changed, 23 insertions(+), 11 deletions(-) diff --git a/userspace/libscap/CMakeLists.txt b/userspace/libscap/CMakeLists.txt index bc99ae1c2..87a1dea85 100644 --- a/userspace/libscap/CMakeLists.txt +++ b/userspace/libscap/CMakeLists.txt @@ -34,13 +34,27 @@ if(CMAKE_SYSTEM_NAME MATCHES "Linux") set(PROBE_VERSION "${SYSDIG_VERSION}") endif() if(NOT DEFINED PROBE_NAME) - set(PROBE_NAME "sysdig-probe") + set(PROBE_NAME "scap") endif() if(NOT DEFINED PROBE_DEVICE_NAME) - set(PROBE_DEVICE_NAME "sysdig") + set(PROBE_DEVICE_NAME "scap") endif() + + if(NOT DEFINED PROBE_ENV_VAR_NAME) + string(TOUPPER PROBE_DEVICE_NAME PROBE_ENV_VAR_NAME) + endif() +endif() + +if(NOT DEFINED PROBE_ENV_VAR_NAME) + set(PROBE_ENV_VAR_NAME "BPF_PROBE") +endif() +add_definitions(-DPROBE_ENV_VAR_NAME="${PROBE_ENV_VAR_NAME}") + +if(NOT DEFINED SCAP_HOST_ROOT_ENV_VAR_NAME) + set(PROBE_ENV_VAR_NAME "HOST_ROOT") endif() +add_definitions(-DSCAP_HOST_ROOT_ENV_VAR_NAME="${SCAP_HOST_ROOT_ENV_VAR_NAME}") if(CYGWIN) include_directories("${WIN_HAL_INCLUDE}") diff --git a/userspace/libscap/scap.c b/userspace/libscap/scap.c index d3d89ac14..872d84d71 100644 --- a/userspace/libscap/scap.c +++ b/userspace/libscap/scap.c @@ -56,8 +56,6 @@ limitations under the License. //#define NDEBUG #include -static const char *SYSDIG_BPF_PROBE_ENV = "FALCO_BPF_PROBE"; - // // Probe version string size // @@ -197,7 +195,7 @@ scap_t* scap_open_live_int(char *error, int32_t *rc, // While in theory we could always rely on the scap caller to properly // set a BPF probe from the environment variable, it's in practice easier // to do one more check here in scap so we don't have to repeat the logic - // in all the possible users of the libraries (sysdig, csysdig, dragent, ...) + // in all the possible users of the libraries // if(!bpf_probe) { @@ -220,7 +218,7 @@ scap_t* scap_open_live_int(char *error, int32_t *rc, return NULL; } - snprintf(buf, sizeof(buf), "%s/.falco/%s-bpf.o", home, PROBE_NAME); + snprintf(buf, sizeof(buf), "%s/.%s/%s-bpf.o", home, PROBE_DEVICE_NAME, PROBE_NAME); bpf_probe = buf; } } @@ -380,7 +378,7 @@ scap_t* scap_open_live_int(char *error, int32_t *rc, else if(errno == EBUSY) { uint32_t curr_max_consumers = get_max_consumers(); - snprintf(error, SCAP_LASTERR_SIZE, "Too many Falco instances attached to device %s. Current value for /sys/module/" PROBE_DEVICE_NAME "/parameters/max_consumers is '%"PRIu32"'.", filename, curr_max_consumers); + snprintf(error, SCAP_LASTERR_SIZE, "Too many consumers attached to device %s. Current value for /sys/module/" PROBE_DEVICE_NAME "/parameters/max_consumers is '%"PRIu32"'.", filename, curr_max_consumers); } else { @@ -471,7 +469,7 @@ scap_t* scap_open_live_int(char *error, int32_t *rc, } // - // Now that sysdig has done all its /proc parsing, start the capture + // Now that /proc parsing has been done, start the capture // if((*rc = scap_start_capture(handle)) != SCAP_SUCCESS) { @@ -676,7 +674,7 @@ scap_t* scap_open_udig_int(char *error, int32_t *rc, } // - // Now that sysdig has done all its /proc parsing, start the capture + // Now that /proc parsing has been done, start the capture // if(udig_begin_capture(handle, error) != SCAP_SUCCESS) { @@ -2509,7 +2507,7 @@ int32_t scap_disable_dynamic_snaplen(scap_t* handle) const char* scap_get_host_root() { - char* p = getenv("HOST_ROOT"); + char* p = getenv(SCAP_HOST_ROOT_ENV_VAR_NAME); static char env_str[SCAP_MAX_PATH_SIZE + 1]; static bool inited = false; if (! inited) { @@ -2710,7 +2708,7 @@ wh_t* scap_get_wmi_handle(scap_t* handle) const char *scap_get_bpf_probe_from_env() { - return getenv(SYSDIG_BPF_PROBE_ENV); + return getenv(PROBE_ENV_VAR_NAME); } bool scap_get_bpf_enabled(scap_t *handle) From 3d3445e13cbf7910ede6ab96edd12c9a5e1d1970 Mon Sep 17 00:00:00 2001 From: Leonardo Grasso Date: Tue, 11 May 2021 11:30:44 +0200 Subject: [PATCH 118/148] update: rename project to "falcosecurity-libs" Signed-off-by: Leonardo Grasso --- CMakeLists.txt | 11 ++++---- cmake/modules/CompilerFlags.cmake | 42 +++++++++++++++---------------- userspace/libscap/CMakeLists.txt | 4 +-- 3 files changed, 28 insertions(+), 29 deletions(-) diff --git a/CMakeLists.txt b/CMakeLists.txt index ad8e0a428..3048c028d 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -27,13 +27,12 @@ if(EXISTS ${CMAKE_CURRENT_BINARY_DIR}/CMakeLists.txt) " 4. Run cmake from the build directory. ex: cmake ..\n" " 5. Run make from the build directory. ex: make\n" "Full paste-able example:\n" - "( rm -f CMakeCache.txt; mkdir build; cd build; cmake ..; make )\n" - "The following wiki page has more information on manually building sysdig: http://bit.ly/1oJ84UI") + "( rm -f CMakeCache.txt; mkdir build; cd build; cmake ..; make )") endif() cmake_minimum_required(VERSION 2.8.2) -project(sysdig) +project(falcosecurity-libs) option(MINIMAL_BUILD "Produce a minimal build with only the essential features (no eBPF probe driver, no kubernetes, no mesos, no marathon and no container metadata)" OFF) option(MUSL_OPTIMIZED_BUILD "Enable if you want a musl optimized build" OFF) @@ -42,15 +41,15 @@ option(MUSL_OPTIMIZED_BUILD "Enable if you want a musl optimized build" OFF) list(APPEND CMAKE_MODULE_PATH "${CMAKE_CURRENT_SOURCE_DIR}/cmake/modules") -if(NOT DEFINED SYSDIG_VERSION) - set(SYSDIG_VERSION "0.1.1dev") +if(NOT DEFINED FALCOSECURITY_LIBS_VERSION) + set(FALCOSECURITY_LIBS_VERSION "0.1.1dev") endif() if(NOT CMAKE_BUILD_TYPE) SET(CMAKE_BUILD_TYPE Release) endif() -set(PACKAGE_NAME "sysdig") +set(PACKAGE_NAME "libs") include(CheckSymbolExists) check_symbol_exists(strlcpy "string.h" HAVE_STRLCPY) diff --git a/cmake/modules/CompilerFlags.cmake b/cmake/modules/CompilerFlags.cmake index ec4b4980f..46de869ce 100644 --- a/cmake/modules/CompilerFlags.cmake +++ b/cmake/modules/CompilerFlags.cmake @@ -8,31 +8,31 @@ endif() if(NOT WIN32) - set(SYSDIG_COMMON_FLAGS "-Wall -ggdb") - set(SYSDIG_DEBUG_FLAGS "-D_DEBUG") - set(SYSDIG_RELEASE_FLAGS "-O3 -fno-strict-aliasing -DNDEBUG") + set(FALCOSECURITY_LIBS_COMMON_FLAGS "-Wall -ggdb") + set(FALCOSECURITY_LIBS_DEBUG_FLAGS "-D_DEBUG") + set(FALCOSECURITY_LIBS_RELEASE_FLAGS "-O3 -fno-strict-aliasing -DNDEBUG") if(MINIMAL_BUILD) - set(SYSDIG_COMMON_FLAGS "${SYSDIG_COMMON_FLAGS} -DMINIMAL_BUILD") + set(FALCOSECURITY_LIBS_COMMON_FLAGS "${FALCOSECURITY_LIBS_COMMON_FLAGS} -DMINIMAL_BUILD") endif() if(MUSL_OPTIMIZED_BUILD) - set(SYSDIG_COMMON_FLAGS "${SYSDIG_COMMON_FLAGS} -static -Os") + set(FALCOSECURITY_LIBS_COMMON_FLAGS "${FALCOSECURITY_LIBS_COMMON_FLAGS} -static -Os") endif() if(BUILD_WARNINGS_AS_ERRORS) set(CMAKE_SUPPRESSED_WARNINGS "-Wno-unused-parameter -Wno-missing-field-initializers -Wno-sign-compare -Wno-type-limits -Wno-implicit-fallthrough -Wno-format-truncation") - set(SYSDIG_COMMON_FLAGS "${SYSDIG_COMMON_FLAGS} -Wextra -Werror ${CMAKE_SUPPRESSED_WARNINGS}") + set(FALCOSECURITY_LIBS_COMMON_FLAGS "${FALCOSECURITY_LIBS_COMMON_FLAGS} -Wextra -Werror ${CMAKE_SUPPRESSED_WARNINGS}") endif() - set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} ${SYSDIG_COMMON_FLAGS}") - set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} ${SYSDIG_COMMON_FLAGS} -std=c++0x") + set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} ${FALCOSECURITY_LIBS_COMMON_FLAGS}") + set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} ${FALCOSECURITY_LIBS_COMMON_FLAGS} -std=c++0x") - set(CMAKE_C_FLAGS_DEBUG "${SYSDIG_DEBUG_FLAGS}") - set(CMAKE_CXX_FLAGS_DEBUG "${SYSDIG_DEBUG_FLAGS}") + set(CMAKE_C_FLAGS_DEBUG "${FALCOSECURITY_LIBS_DEBUG_FLAGS}") + set(CMAKE_CXX_FLAGS_DEBUG "${FALCOSECURITY_LIBS_DEBUG_FLAGS}") - set(CMAKE_C_FLAGS_RELEASE "${SYSDIG_RELEASE_FLAGS}") - set(CMAKE_CXX_FLAGS_RELEASE "${SYSDIG_RELEASE_FLAGS}") + set(CMAKE_C_FLAGS_RELEASE "${FALCOSECURITY_LIBS_RELEASE_FLAGS}") + set(CMAKE_CXX_FLAGS_RELEASE "${FALCOSECURITY_LIBS_RELEASE_FLAGS}") if(CMAKE_SYSTEM_NAME MATCHES "Linux") add_definitions(-DHAS_CAPTURE) @@ -41,18 +41,18 @@ if(NOT WIN32) else() set(MINIMAL_BUILD ON) - set(SYSDIG_COMMON_FLAGS "-D_CRT_SECURE_NO_WARNINGS -DWIN32 -DMINIMAL_BUILD /EHsc /W3 /Zi") - set(SYSDIG_DEBUG_FLAGS "/MTd /Od") - set(SYSDIG_RELEASE_FLAGS "/MT") + set(FALCOSECURITY_LIBS_COMMON_FLAGS "-D_CRT_SECURE_NO_WARNINGS -DWIN32 -DMINIMAL_BUILD /EHsc /W3 /Zi") + set(FALCOSECURITY_LIBS_DEBUG_FLAGS "/MTd /Od") + set(FALCOSECURITY_LIBS_RELEASE_FLAGS "/MT") - set(CMAKE_C_FLAGS "${SYSDIG_COMMON_FLAGS}") - set(CMAKE_CXX_FLAGS "${SYSDIG_COMMON_FLAGS}") + set(CMAKE_C_FLAGS "${FALCOSECURITY_LIBS_COMMON_FLAGS}") + set(CMAKE_CXX_FLAGS "${FALCOSECURITY_LIBS_COMMON_FLAGS}") - set(CMAKE_C_FLAGS_DEBUG "${SYSDIG_DEBUG_FLAGS}") - set(CMAKE_CXX_FLAGS_DEBUG "${SYSDIG_DEBUG_FLAGS}") + set(CMAKE_C_FLAGS_DEBUG "${FALCOSECURITY_LIBS_DEBUG_FLAGS}") + set(CMAKE_CXX_FLAGS_DEBUG "${FALCOSECURITY_LIBS_DEBUG_FLAGS}") - set(CMAKE_C_FLAGS_RELEASE "${SYSDIG_RELEASE_FLAGS}") - set(CMAKE_CXX_FLAGS_RELEASE "${SYSDIG_RELEASE_FLAGS}") + set(CMAKE_C_FLAGS_RELEASE "${FALCOSECURITY_LIBS_RELEASE_FLAGS}") + set(CMAKE_CXX_FLAGS_RELEASE "${FALCOSECURITY_LIBS_RELEASE_FLAGS}") add_definitions(-DHAS_CAPTURE) diff --git a/userspace/libscap/CMakeLists.txt b/userspace/libscap/CMakeLists.txt index 87a1dea85..077a497e1 100644 --- a/userspace/libscap/CMakeLists.txt +++ b/userspace/libscap/CMakeLists.txt @@ -27,11 +27,11 @@ add_definitions(-DPLATFORM_NAME="${CMAKE_SYSTEM_NAME}") if(CMAKE_SYSTEM_NAME MATCHES "Linux") if(CMAKE_BUILD_TYPE STREQUAL "Debug") - set(KBUILD_FLAGS "${SYSDIG_DEBUG_FLAGS}") + set(KBUILD_FLAGS "${FALCOSECURITY_LIBS_DEBUG_FLAGS}") endif() if(NOT DEFINED PROBE_VERSION) - set(PROBE_VERSION "${SYSDIG_VERSION}") + set(PROBE_VERSION "${FALCOSECURITY_LIBS_VERSION}") endif() if(NOT DEFINED PROBE_NAME) set(PROBE_NAME "scap") From dfd4f8b4427f1a420fdceb716288f1ab9c70263f Mon Sep 17 00:00:00 2001 From: Leonardo Grasso Date: Thu, 1 Jul 2021 15:27:28 +0200 Subject: [PATCH 119/148] fix(userspace/libscap): correct SCAP_HOST_ROOT_ENV_VAR_NAME defalt value Co-authored-by: Grzegorz Nosek Signed-off-by: Leonardo Grasso --- userspace/libscap/CMakeLists.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/userspace/libscap/CMakeLists.txt b/userspace/libscap/CMakeLists.txt index 077a497e1..9ec766a2b 100644 --- a/userspace/libscap/CMakeLists.txt +++ b/userspace/libscap/CMakeLists.txt @@ -52,7 +52,7 @@ endif() add_definitions(-DPROBE_ENV_VAR_NAME="${PROBE_ENV_VAR_NAME}") if(NOT DEFINED SCAP_HOST_ROOT_ENV_VAR_NAME) - set(PROBE_ENV_VAR_NAME "HOST_ROOT") + set(SCAP_HOST_ROOT_ENV_VAR_NAME "HOST_ROOT") endif() add_definitions(-DSCAP_HOST_ROOT_ENV_VAR_NAME="${SCAP_HOST_ROOT_ENV_VAR_NAME}") From 081f4b8f409aad4d235fd97c3b031591e84c26af Mon Sep 17 00:00:00 2001 From: Leonardo Grasso Date: Thu, 1 Jul 2021 17:02:39 +0200 Subject: [PATCH 120/148] build(userspace/libscap): `SCAP_BPF_PROBE_ENV_VAR_NAME` cmake var Signed-off-by: Leonardo Grasso --- userspace/libscap/CMakeLists.txt | 10 +++------- userspace/libscap/scap.c | 2 +- 2 files changed, 4 insertions(+), 8 deletions(-) diff --git a/userspace/libscap/CMakeLists.txt b/userspace/libscap/CMakeLists.txt index 9ec766a2b..d74b745b9 100644 --- a/userspace/libscap/CMakeLists.txt +++ b/userspace/libscap/CMakeLists.txt @@ -40,16 +40,12 @@ if(CMAKE_SYSTEM_NAME MATCHES "Linux") if(NOT DEFINED PROBE_DEVICE_NAME) set(PROBE_DEVICE_NAME "scap") endif() - - if(NOT DEFINED PROBE_ENV_VAR_NAME) - string(TOUPPER PROBE_DEVICE_NAME PROBE_ENV_VAR_NAME) - endif() endif() -if(NOT DEFINED PROBE_ENV_VAR_NAME) - set(PROBE_ENV_VAR_NAME "BPF_PROBE") +if(NOT DEFINED SCAP_BPF_PROBE_ENV_VAR_NAME) + set(SCAP_BPF_PROBE_ENV_VAR_NAME "BPF_PROBE") endif() -add_definitions(-DPROBE_ENV_VAR_NAME="${PROBE_ENV_VAR_NAME}") +add_definitions(-DSCAP_BPF_PROBE_ENV_VAR_NAME="${SCAP_BPF_PROBE_ENV_VAR_NAME}") if(NOT DEFINED SCAP_HOST_ROOT_ENV_VAR_NAME) set(SCAP_HOST_ROOT_ENV_VAR_NAME "HOST_ROOT") diff --git a/userspace/libscap/scap.c b/userspace/libscap/scap.c index 872d84d71..3a2a3f1a8 100644 --- a/userspace/libscap/scap.c +++ b/userspace/libscap/scap.c @@ -2708,7 +2708,7 @@ wh_t* scap_get_wmi_handle(scap_t* handle) const char *scap_get_bpf_probe_from_env() { - return getenv(PROBE_ENV_VAR_NAME); + return getenv(SCAP_BPF_PROBE_ENV_VAR_NAME); } bool scap_get_bpf_enabled(scap_t *handle) From 9c5422657efdaad755dd4f98dbe6289199840a5e Mon Sep 17 00:00:00 2001 From: Leonardo Grasso Date: Thu, 1 Jul 2021 17:16:11 +0200 Subject: [PATCH 121/148] chore(userspace/libscap): generate module name from `PROBE_NAME` Co-authored-by: Grzegorz Nosek Signed-off-by: Leonardo Grasso --- userspace/libscap/CMakeLists.txt | 9 +++++++-- userspace/libscap/scap.c | 4 ++-- 2 files changed, 9 insertions(+), 4 deletions(-) diff --git a/userspace/libscap/CMakeLists.txt b/userspace/libscap/CMakeLists.txt index d74b745b9..44ce0656f 100644 --- a/userspace/libscap/CMakeLists.txt +++ b/userspace/libscap/CMakeLists.txt @@ -33,13 +33,18 @@ if(CMAKE_SYSTEM_NAME MATCHES "Linux") if(NOT DEFINED PROBE_VERSION) set(PROBE_VERSION "${FALCOSECURITY_LIBS_VERSION}") endif() + if(NOT DEFINED PROBE_NAME) set(PROBE_NAME "scap") endif() - + if(NOT DEFINED PROBE_DEVICE_NAME) - set(PROBE_DEVICE_NAME "scap") + set(PROBE_DEVICE_NAME "${PROBE_NAME}") endif() + + string(REPLACE "-" "_" SCAP_PROBE_MODULE_NAME "${PROBE_NAME}") + add_definitions(-DSCAP_PROBE_MODULE_NAME="${SCAP_PROBE_MODULE_NAME}") + endif() if(NOT DEFINED SCAP_BPF_PROBE_ENV_VAR_NAME) diff --git a/userspace/libscap/scap.c b/userspace/libscap/scap.c index 3a2a3f1a8..225382e19 100644 --- a/userspace/libscap/scap.c +++ b/userspace/libscap/scap.c @@ -144,7 +144,7 @@ static uint32_t get_max_consumers() { #ifndef _WIN32 uint32_t max; - FILE *pfile = fopen("/sys/module/" PROBE_DEVICE_NAME "/parameters/max_consumers", "r"); + FILE *pfile = fopen("/sys/module/" SCAP_PROBE_MODULE_NAME "/parameters/max_consumers", "r"); if(pfile != NULL) { int w = fscanf(pfile, "%"PRIu32, &max); @@ -378,7 +378,7 @@ scap_t* scap_open_live_int(char *error, int32_t *rc, else if(errno == EBUSY) { uint32_t curr_max_consumers = get_max_consumers(); - snprintf(error, SCAP_LASTERR_SIZE, "Too many consumers attached to device %s. Current value for /sys/module/" PROBE_DEVICE_NAME "/parameters/max_consumers is '%"PRIu32"'.", filename, curr_max_consumers); + snprintf(error, SCAP_LASTERR_SIZE, "Too many consumers attached to device %s. Current value for /sys/module/" SCAP_PROBE_MODULE_NAME "/parameters/max_consumers is '%"PRIu32"'.", filename, curr_max_consumers); } else { From 740f36514b5b0b9d47093d1d8e461990c611e9a7 Mon Sep 17 00:00:00 2001 From: Leonardo Grasso Date: Thu, 1 Jul 2021 17:38:33 +0200 Subject: [PATCH 122/148] build(userspace/libscap): `PROBE_BPF_FILEPATH` cmake var Signed-off-by: Leonardo Grasso --- userspace/libscap/CMakeLists.txt | 5 +++++ userspace/libscap/scap.c | 2 +- 2 files changed, 6 insertions(+), 1 deletion(-) diff --git a/userspace/libscap/CMakeLists.txt b/userspace/libscap/CMakeLists.txt index 44ce0656f..e32e761f5 100644 --- a/userspace/libscap/CMakeLists.txt +++ b/userspace/libscap/CMakeLists.txt @@ -45,6 +45,11 @@ if(CMAKE_SYSTEM_NAME MATCHES "Linux") string(REPLACE "-" "_" SCAP_PROBE_MODULE_NAME "${PROBE_NAME}") add_definitions(-DSCAP_PROBE_MODULE_NAME="${SCAP_PROBE_MODULE_NAME}") + if(NOT DEFINED SCAP_PROBE_BPF_FILEPATH) + # note that the home folder is prepended by scap at runtime + set(SCAP_PROBE_BPF_FILEPATH ".${PROBE_NAME}/${PROBE_NAME}-bpf.o") + endif() + add_definitions(-DSCAP_PROBE_BPF_FILEPATH="${SCAP_PROBE_BPF_FILEPATH}") endif() if(NOT DEFINED SCAP_BPF_PROBE_ENV_VAR_NAME) diff --git a/userspace/libscap/scap.c b/userspace/libscap/scap.c index 225382e19..07c5c2aa3 100644 --- a/userspace/libscap/scap.c +++ b/userspace/libscap/scap.c @@ -218,7 +218,7 @@ scap_t* scap_open_live_int(char *error, int32_t *rc, return NULL; } - snprintf(buf, sizeof(buf), "%s/.%s/%s-bpf.o", home, PROBE_DEVICE_NAME, PROBE_NAME); + snprintf(buf, sizeof(buf), "%s/%s", home, SCAP_PROBE_BPF_FILEPATH); bpf_probe = buf; } } From 06b36b7bb2866db44a4e39125d90131faa207205 Mon Sep 17 00:00:00 2001 From: Leonardo Grasso Date: Thu, 22 Jul 2021 10:56:57 +0200 Subject: [PATCH 123/148] docs(userspace/libsinsp): update doxygen headers Signed-off-by: Leonardo Grasso --- userspace/libscap/doxygen/header.html | 2 +- userspace/libsinsp/doxygen/header.html | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/userspace/libscap/doxygen/header.html b/userspace/libscap/doxygen/header.html index 6ca6b86ba..a2668932c 100644 --- a/userspace/libscap/doxygen/header.html +++ b/userspace/libscap/doxygen/header.html @@ -1,6 +1,6 @@ --- layout: default -title: sysdig | libscap +title: falcosecurity | libscap --- diff --git a/userspace/libsinsp/doxygen/header.html b/userspace/libsinsp/doxygen/header.html index 68d51a131..aa5c92f20 100644 --- a/userspace/libsinsp/doxygen/header.html +++ b/userspace/libsinsp/doxygen/header.html @@ -1,6 +1,6 @@ --- layout: default -title: sysdig | libsinsp +title: falcosecurity | libsinsp --- From 9f6b4f2da5af4984994227b74b955ed59ba93b13 Mon Sep 17 00:00:00 2001 From: Leonardo Grasso Date: Thu, 22 Jul 2021 10:58:04 +0200 Subject: [PATCH 124/148] chore(userspace): clean up comments Signed-off-by: Leonardo Grasso --- userspace/chisel/chisel.h | 2 +- userspace/chisel/chisel_fields_info.h | 1 - userspace/libsinsp/k8s_component.cpp | 1 - userspace/libsinsp/tracer_emitter.h | 2 +- 4 files changed, 2 insertions(+), 4 deletions(-) diff --git a/userspace/chisel/chisel.h b/userspace/chisel/chisel.h index ef621b740..2efc76e42 100644 --- a/userspace/chisel/chisel.h +++ b/userspace/chisel/chisel.h @@ -39,7 +39,7 @@ typedef struct lua_State lua_State; */ /*! - \brief This is the class that compiles and runs sysdig-type filters. + \brief This is the class that compiles and runs filters. */ typedef struct chiseldir_info { diff --git a/userspace/chisel/chisel_fields_info.h b/userspace/chisel/chisel_fields_info.h index 1ca643a64..01729a751 100644 --- a/userspace/chisel/chisel_fields_info.h +++ b/userspace/chisel/chisel_fields_info.h @@ -1,7 +1,6 @@ /* Copyright (C) 2021 The Falco Authors. -This file is part of sysdig. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. diff --git a/userspace/libsinsp/k8s_component.cpp b/userspace/libsinsp/k8s_component.cpp index 76902bc9a..202857d87 100644 --- a/userspace/libsinsp/k8s_component.cpp +++ b/userspace/libsinsp/k8s_component.cpp @@ -917,7 +917,6 @@ bool k8s_event_t::update(const Json::Value& item, k8s_state_t& state) user_event_logger::log(evt, severity); - // TODO: sysdig capture? #endif // _WIN32 return true; diff --git a/userspace/libsinsp/tracer_emitter.h b/userspace/libsinsp/tracer_emitter.h index f8d8a4e5f..d5c6a3fc6 100644 --- a/userspace/libsinsp/tracer_emitter.h +++ b/userspace/libsinsp/tracer_emitter.h @@ -17,7 +17,7 @@ limitations under the License. #pragma once #include -// This class allows the caller to output sysdig tracers +// This class allows the caller to output tracers // to /dev/null. class tracer_emitter { From d3a3d5bf0f4afcbbbf22fbdcf98b5527e03c18e1 Mon Sep 17 00:00:00 2001 From: Leonardo Grasso Date: Mon, 15 Nov 2021 09:49:14 +0100 Subject: [PATCH 125/148] build: correct driver package name Signed-off-by: Leonardo Grasso --- CMakeLists.txt | 2 +- driver/CMakeLists.txt | 4 ++-- driver/bpf/CMakeLists.txt | 4 ++-- 3 files changed, 5 insertions(+), 5 deletions(-) diff --git a/CMakeLists.txt b/CMakeLists.txt index 3048c028d..b0544c90c 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -49,7 +49,7 @@ if(NOT CMAKE_BUILD_TYPE) SET(CMAKE_BUILD_TYPE Release) endif() -set(PACKAGE_NAME "libs") +set(DRIVER_PACKAGE_NAME "scap") include(CheckSymbolExists) check_symbol_exists(strlcpy "string.h" HAVE_STRLCPY) diff --git a/driver/CMakeLists.txt b/driver/CMakeLists.txt index 14f086fcf..b446ab4d8 100644 --- a/driver/CMakeLists.txt +++ b/driver/CMakeLists.txt @@ -103,8 +103,8 @@ if(ENABLE_DKMS) ${CMAKE_CURRENT_BINARY_DIR}/src/dkms.conf ${CMAKE_CURRENT_BINARY_DIR}/src/driver_config.h ${DRIVER_SOURCES} - DESTINATION "src/${PACKAGE_NAME}-${PROBE_VERSION}" - COMPONENT agent-kmodule) + DESTINATION "src/${DRIVER_PACKAGE_NAME}-${PROBE_VERSION}" + COMPONENT scap-driver) endif() diff --git a/driver/bpf/CMakeLists.txt b/driver/bpf/CMakeLists.txt index e55761d4f..99321d6d0 100644 --- a/driver/bpf/CMakeLists.txt +++ b/driver/bpf/CMakeLists.txt @@ -29,5 +29,5 @@ install(FILES quirks.h ring_helpers.h types.h - DESTINATION "src/${PACKAGE_NAME}-${PROBE_VERSION}/bpf" - COMPONENT agent-kmodule) + DESTINATION "src/${DRIVER_PACKAGE_NAME}-${PROBE_VERSION}/bpf" + COMPONENT scap-driver) From a743aa4ab28e4bd82c73f74be4f866911a4be0fa Mon Sep 17 00:00:00 2001 From: Leonardo Grasso Date: Mon, 15 Nov 2021 10:16:31 +0100 Subject: [PATCH 126/148] refactor: correct var naming Signed-off-by: Leonardo Grasso --- userspace/libsinsp/parsers.cpp | 20 ++++++++++---------- userspace/libsinsp/sinsp.cpp | 4 ++-- userspace/libsinsp/sinsp.h | 2 +- userspace/libsinsp/threadinfo.cpp | 10 +++++----- 4 files changed, 18 insertions(+), 18 deletions(-) diff --git a/userspace/libsinsp/parsers.cpp b/userspace/libsinsp/parsers.cpp index 1e7b4b470..64e288fdf 100644 --- a/userspace/libsinsp/parsers.cpp +++ b/userspace/libsinsp/parsers.cpp @@ -130,20 +130,20 @@ void sinsp_parser::process_event(sinsp_evt *evt) reset(evt); // - // When debug mode is not enabled, filter out events about sysdig itself + // When debug mode is not enabled, filter out events about itself // #if defined(HAS_CAPTURE) if(is_live && !m_inspector->is_debug_enabled()) { - if(evt->get_tid() == m_inspector->m_sysdig_pid && - etype != PPME_SCHEDSWITCH_1_E && - etype != PPME_SCHEDSWITCH_6_E && - etype != PPME_DROP_E && - etype != PPME_DROP_X && - etype != PPME_SYSDIGEVENT_E && - etype != PPME_PROCINFO_E && - etype != PPME_CPU_HOTPLUG_E && - m_inspector->m_sysdig_pid) + if(evt->get_tid() == m_inspector->m_self_pid && + etype != PPME_SCHEDSWITCH_1_E && + etype != PPME_SCHEDSWITCH_6_E && + etype != PPME_DROP_E && + etype != PPME_DROP_X && + etype != PPME_SYSDIGEVENT_E && + etype != PPME_PROCINFO_E && + etype != PPME_CPU_HOTPLUG_E && + m_inspector->m_self_pid) { evt->m_filtered_out = true; return; diff --git a/userspace/libsinsp/sinsp.cpp b/userspace/libsinsp/sinsp.cpp index a8285e97a..1bb17ca1e 100644 --- a/userspace/libsinsp/sinsp.cpp +++ b/userspace/libsinsp/sinsp.cpp @@ -128,7 +128,7 @@ sinsp::sinsp(bool static_container, const std::string static_id, const std::stri m_print_container_data = false; #if defined(HAS_CAPTURE) - m_sysdig_pid = getpid(); + m_self_pid = getpid(); #endif uint32_t evlen = sizeof(scap_evt) + 2 * sizeof(uint16_t) + 2 * sizeof(uint64_t); @@ -431,7 +431,7 @@ void sinsp::init() #if defined(HAS_CAPTURE) if(m_mode == SCAP_MODE_LIVE) { - if(scap_getpid_global(m_h, &m_sysdig_pid) != SCAP_SUCCESS) + if(scap_getpid_global(m_h, &m_self_pid) != SCAP_SUCCESS) { ASSERT(false); } diff --git a/userspace/libsinsp/sinsp.h b/userspace/libsinsp/sinsp.h index a9612a5cc..4928f7afa 100644 --- a/userspace/libsinsp/sinsp.h +++ b/userspace/libsinsp/sinsp.h @@ -1225,7 +1225,7 @@ VISIBILITY_PRIVATE static unsigned int m_num_possible_cpus; #if defined(HAS_CAPTURE) - int64_t m_sysdig_pid; + int64_t m_self_pid; #endif // Any thread with a comm in this set will not have its events diff --git a/userspace/libsinsp/threadinfo.cpp b/userspace/libsinsp/threadinfo.cpp index 1bb3498a1..77d015267 100644 --- a/userspace/libsinsp/threadinfo.cpp +++ b/userspace/libsinsp/threadinfo.cpp @@ -1263,11 +1263,11 @@ bool sinsp_thread_manager::add_thread(sinsp_threadinfo *threadinfo, bool from_sc m_last_tinfo.reset(); - if (m_threadtable.size() >= m_max_thread_table_size + if(m_threadtable.size() >= m_max_thread_table_size #if defined(HAS_CAPTURE) - && threadinfo->m_pid != m_inspector->m_sysdig_pid + && threadinfo->m_pid != m_inspector->m_self_pid #endif - ) + ) { // rate limit messages to avoid spamming the logs if (m_n_drops % m_max_thread_table_size == 0) @@ -1668,9 +1668,9 @@ threadinfo_map_t::ptr_t sinsp_thread_manager::get_thread_ref(int64_t tid, bool q if(!sinsp_proc && query_os_if_not_found && (m_threadtable.size() < m_max_thread_table_size #if defined(HAS_CAPTURE) - || tid == m_inspector->m_sysdig_pid + || tid == m_inspector->m_self_pid #endif - )) + )) { // Certain code paths can lead to this point from scap_open() (incomplete example: // scap_proc_scan_proc_dir() -> resolve_container() -> get_env()). Adding a From 44a457d8ef127107b50ac05f745ee78079aa4d2b Mon Sep 17 00:00:00 2001 From: Mark Stemm Date: Wed, 17 Nov 2021 17:28:35 -0800 Subject: [PATCH 127/148] Define __STDC_FORMAT_MACROS for inttypes.h (older g++) When including inttypes.h, define __STDC_FORMAT_MACROS to ensure that the PRIu32 defines are included. With newer g++ versions this happens automatically, but with older g++ versions, it's still required. Signed-off-by: Mark Stemm --- userspace/libsinsp/plugin.cpp | 2 ++ 1 file changed, 2 insertions(+) diff --git a/userspace/libsinsp/plugin.cpp b/userspace/libsinsp/plugin.cpp index fe466f35d..d8be5fb73 100755 --- a/userspace/libsinsp/plugin.cpp +++ b/userspace/libsinsp/plugin.cpp @@ -17,6 +17,8 @@ limitations under the License. #ifndef _WIN32 #include +// This makes inttypes.h define PRIu32 (ISO C99 plus older g++ versions) +#define __STDC_FORMAT_MACROS #include #include #include From de277bbc1e7a92edc84db9a925fb3e12c7b1cb07 Mon Sep 17 00:00:00 2001 From: Grzegorz Nosek Date: Thu, 18 Nov 2021 17:05:00 +0100 Subject: [PATCH 128/148] fix(build): Use correct package name in DKMS config file Commit d3a3d5bf0f4afcbbbf22fbdcf98b5527e03c18e1 renamed PACKAGE_NAME to DRIVER_PACKAGE_NAME. While doing so, it missed the template substitution in driver/dkms.conf.in Signed-off-by: Grzegorz Nosek --- driver/dkms.conf.in | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/driver/dkms.conf.in b/driver/dkms.conf.in index afa8ee748..e83cc37d9 100644 --- a/driver/dkms.conf.in +++ b/driver/dkms.conf.in @@ -1,4 +1,4 @@ -PACKAGE_NAME="@PACKAGE_NAME@" +PACKAGE_NAME="@DRIVER_PACKAGE_NAME@" PACKAGE_VERSION="@PROBE_VERSION@" BUILT_MODULE_NAME[0]="@PROBE_NAME@" DEST_MODULE_LOCATION[0]="/kernel/extra" From 99e9a3bafaca89a2d1ff220fa76a161cc05dd9a5 Mon Sep 17 00:00:00 2001 From: Federico Di Pierro Date: Fri, 19 Nov 2021 14:37:25 +0100 Subject: [PATCH 129/148] fix(driver): small fix for eBPF probe on amznlinux2 kernels. Signed-off-by: Federico Di Pierro --- driver/bpf/filler_helpers.h | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/driver/bpf/filler_helpers.h b/driver/bpf/filler_helpers.h index 48a143905..7b2538637 100644 --- a/driver/bpf/filler_helpers.h +++ b/driver/bpf/filler_helpers.h @@ -813,6 +813,10 @@ static __always_inline int __bpf_val_to_ring(struct filler_data *data, if (!data->curarg_already_on_frame) { volatile u16 read_size = len; + curoff_bounded = data->state->tail_ctx.curoff & SCRATCH_SIZE_HALF; + if (data->state->tail_ctx.curoff > SCRATCH_SIZE_HALF) + return PPM_FAILURE_BUFFER_FULL; + #ifdef BPF_FORBIDS_ZERO_ACCESS if (read_size) if (bpf_probe_read(&data->buf[curoff_bounded], From a89234bad257d4af6469697bd92509c0c3ff2039 Mon Sep 17 00:00:00 2001 From: Federico Di Pierro Date: Tue, 23 Nov 2021 17:17:34 +0100 Subject: [PATCH 130/148] fix(build): update grpc cmake module to use find_package() when building using system library. This fixes the linking of libraries at the end of Falco build. Signed-off-by: Federico Di Pierro Co-authored-by: Leonardo Grasso --- cmake/modules/grpc.cmake | 35 +++++++++++++++-------------------- 1 file changed, 15 insertions(+), 20 deletions(-) diff --git a/cmake/modules/grpc.cmake b/cmake/modules/grpc.cmake index ac3b536eb..12f217063 100644 --- a/cmake/modules/grpc.cmake +++ b/cmake/modules/grpc.cmake @@ -3,31 +3,26 @@ option(USE_BUNDLED_GRPC "Enable building of the bundled grpc" ${USE_BUNDLED_DEPS if(GRPC_INCLUDE) # we already have grpc elseif(NOT USE_BUNDLED_GRPC) - find_library(GPR_LIB NAMES gpr) - if(GPR_LIB) - message(STATUS "Found gpr lib: ${GPR_LIB}") - else() - message(FATAL_ERROR "Couldn't find system gpr") + # gRPC + find_package(gRPC CONFIG REQUIRED) + message(STATUS "Using gRPC ${gRPC_VERSION}") + set(GPR_LIB gRPC::gpr) + set(GRPC_LIB gRPC::grpc) + set(GRPCPP_LIB gRPC::grpc++) + + # # gRPC C++ plugin + get_target_property(GRPC_CPP_PLUGIN gRPC::grpc_cpp_plugin LOCATION) + if(NOT GRPC_CPP_PLUGIN) + message(FATAL_ERROR "System grpc_cpp_plugin not found") endif() + + # gRPC include dir + properly handle grpc{++,pp} + get_target_property(GRPC_INCLUDE gRPC::grpc++ INTERFACE_INCLUDE_DIRECTORIES) find_path(GRPCXX_INCLUDE NAMES grpc++/grpc++.h) - if(GRPCXX_INCLUDE) - set(GRPC_INCLUDE ${GRPCXX_INCLUDE}) - else() + if(NOT GRPCXX_INCLUDE) find_path(GRPCPP_INCLUDE NAMES grpcpp/grpcpp.h) - set(GRPC_INCLUDE ${GRPCPP_INCLUDE}) add_definitions(-DGRPC_INCLUDE_IS_GRPCPP=1) endif() - find_library(GRPC_LIB NAMES grpc) - find_library(GRPCPP_LIB NAMES grpc++) - if(GRPC_INCLUDE AND GRPC_LIB AND GRPCPP_LIB) - message(STATUS "Found grpc: include: ${GRPC_INCLUDE}, C lib: ${GRPC_LIB}, C++ lib: ${GRPCPP_LIB}") - else() - message(FATAL_ERROR "Couldn't find system grpc") - endif() - find_program(GRPC_CPP_PLUGIN grpc_cpp_plugin) - if(NOT GRPC_CPP_PLUGIN) - message(FATAL_ERROR "System grpc_cpp_plugin not found") - endif() else() include(cares) include(protobuf) From c489384e7a1e698a7765708514219be2ff155179 Mon Sep 17 00:00:00 2001 From: Federico Di Pierro Date: Wed, 24 Nov 2021 11:27:04 +0100 Subject: [PATCH 131/148] update(build): fallback to manually find libraries when grpc config cmake module is not found on system. Ubuntu focal is not shipping the module. Avoid breaking CI and builds on it. Signed-off-by: Federico Di Pierro --- cmake/modules/grpc.cmake | 62 +++++++++++++++++++++++++++++----------- 1 file changed, 46 insertions(+), 16 deletions(-) diff --git a/cmake/modules/grpc.cmake b/cmake/modules/grpc.cmake index 12f217063..51fb76a80 100644 --- a/cmake/modules/grpc.cmake +++ b/cmake/modules/grpc.cmake @@ -4,24 +4,54 @@ if(GRPC_INCLUDE) # we already have grpc elseif(NOT USE_BUNDLED_GRPC) # gRPC - find_package(gRPC CONFIG REQUIRED) - message(STATUS "Using gRPC ${gRPC_VERSION}") - set(GPR_LIB gRPC::gpr) - set(GRPC_LIB gRPC::grpc) - set(GRPCPP_LIB gRPC::grpc++) + find_package(gRPC CONFIG) + if(gRPC_FOUND) + message(STATUS "Using gRPC ${gRPC_VERSION}") + set(GPR_LIB gRPC::gpr) + set(GRPC_LIB gRPC::grpc) + set(GRPCPP_LIB gRPC::grpc++) - # # gRPC C++ plugin - get_target_property(GRPC_CPP_PLUGIN gRPC::grpc_cpp_plugin LOCATION) - if(NOT GRPC_CPP_PLUGIN) - message(FATAL_ERROR "System grpc_cpp_plugin not found") - endif() + # # gRPC C++ plugin + get_target_property(GRPC_CPP_PLUGIN gRPC::grpc_cpp_plugin LOCATION) + if(NOT GRPC_CPP_PLUGIN) + message(FATAL_ERROR "System grpc_cpp_plugin not found") + endif() - # gRPC include dir + properly handle grpc{++,pp} - get_target_property(GRPC_INCLUDE gRPC::grpc++ INTERFACE_INCLUDE_DIRECTORIES) - find_path(GRPCXX_INCLUDE NAMES grpc++/grpc++.h) - if(NOT GRPCXX_INCLUDE) - find_path(GRPCPP_INCLUDE NAMES grpcpp/grpcpp.h) - add_definitions(-DGRPC_INCLUDE_IS_GRPCPP=1) + # gRPC include dir + properly handle grpc{++,pp} + get_target_property(GRPC_INCLUDE gRPC::grpc++ INTERFACE_INCLUDE_DIRECTORIES) + find_path(GRPCXX_INCLUDE NAMES grpc++/grpc++.h) + if(NOT GRPCXX_INCLUDE) + find_path(GRPCPP_INCLUDE NAMES grpcpp/grpcpp.h) + add_definitions(-DGRPC_INCLUDE_IS_GRPCPP=1) + endif() + else() + # Fallback to manually find libraries; + # Some distro, namely Ubuntu focal, do not install gRPC config cmake module + find_library(GPR_LIB NAMES gpr) + if(GPR_LIB) + message(STATUS "Found gpr lib: ${GPR_LIB}") + else() + message(FATAL_ERROR "Couldn't find system gpr") + endif() + find_path(GRPCXX_INCLUDE NAMES grpc++/grpc++.h) + if(GRPCXX_INCLUDE) + set(GRPC_INCLUDE ${GRPCXX_INCLUDE}) + else() + find_path(GRPCPP_INCLUDE NAMES grpcpp/grpcpp.h) + set(GRPC_INCLUDE ${GRPCPP_INCLUDE}) + add_definitions(-DGRPC_INCLUDE_IS_GRPCPP=1) + endif() + find_library(GRPC_LIB NAMES grpc) + find_library(GRPCPP_LIB NAMES grpc++) + if(GRPC_INCLUDE AND GRPC_LIB AND GRPCPP_LIB) + message(STATUS "Found grpc: include: ${GRPC_INCLUDE}, C lib: ${GRPC_LIB}, C++ lib: ${GRPCPP_LIB}") + else() + message(FATAL_ERROR "Couldn't find system grpc") + endif() + find_program(GRPC_CPP_PLUGIN grpc_cpp_plugin) + if(NOT GRPC_CPP_PLUGIN) + message(FATAL_ERROR "System grpc_cpp_plugin not found") + endif() endif() else() include(cares) From d1a2dd9bcd7ff010f5268b30a1fd7a09cebecd7f Mon Sep 17 00:00:00 2001 From: Federico Di Pierro Date: Fri, 26 Nov 2021 09:15:45 +0100 Subject: [PATCH 132/148] Update cmake/modules/grpc.cmake Signed-off-by: Federico Di Pierro Co-authored-by: Grzegorz Nosek --- cmake/modules/grpc.cmake | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/cmake/modules/grpc.cmake b/cmake/modules/grpc.cmake index 51fb76a80..504ea3ba0 100644 --- a/cmake/modules/grpc.cmake +++ b/cmake/modules/grpc.cmake @@ -11,7 +11,7 @@ elseif(NOT USE_BUNDLED_GRPC) set(GRPC_LIB gRPC::grpc) set(GRPCPP_LIB gRPC::grpc++) - # # gRPC C++ plugin + # gRPC C++ plugin get_target_property(GRPC_CPP_PLUGIN gRPC::grpc_cpp_plugin LOCATION) if(NOT GRPC_CPP_PLUGIN) message(FATAL_ERROR "System grpc_cpp_plugin not found") From 8808a06165a546ddd57dc1d4d557da6eeff56b29 Mon Sep 17 00:00:00 2001 From: Federico Di Pierro Date: Mon, 29 Nov 2021 15:39:39 +0100 Subject: [PATCH 133/148] fix(build): check GRPC_INCLUDE_IS_GRPCPP macro using GRPC_INCLUDE as path hint. Signed-off-by: Federico Di Pierro Co-authored-by: Grzegorz Nosek --- cmake/modules/grpc.cmake | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/cmake/modules/grpc.cmake b/cmake/modules/grpc.cmake index 504ea3ba0..ce0646f13 100644 --- a/cmake/modules/grpc.cmake +++ b/cmake/modules/grpc.cmake @@ -19,9 +19,9 @@ elseif(NOT USE_BUNDLED_GRPC) # gRPC include dir + properly handle grpc{++,pp} get_target_property(GRPC_INCLUDE gRPC::grpc++ INTERFACE_INCLUDE_DIRECTORIES) - find_path(GRPCXX_INCLUDE NAMES grpc++/grpc++.h) + find_path(GRPCXX_INCLUDE NAMES grpc++/grpc++.h PATHS ${GRPC_INCLUDE}) if(NOT GRPCXX_INCLUDE) - find_path(GRPCPP_INCLUDE NAMES grpcpp/grpcpp.h) + find_path(GRPCPP_INCLUDE NAMES grpcpp/grpcpp.h PATHS ${GRPC_INCLUDE}) add_definitions(-DGRPC_INCLUDE_IS_GRPCPP=1) endif() else() From 51403ef5893d52771fdb3d149639e9c1579ba422 Mon Sep 17 00:00:00 2001 From: Loris Degioanni Date: Mon, 22 Nov 2021 15:44:45 -0800 Subject: [PATCH 134/148] fixes to make the libs compile on Mac Signed-off-by: Loris Degioanni --- CMakeLists.txt | 1 + cmake/modules/libsinsp.cmake | 16 +++++++++++----- cmake/modules/luajit.cmake | 10 ++++++++++ userspace/common/strlcpy.h | 2 ++ 4 files changed, 24 insertions(+), 5 deletions(-) diff --git a/CMakeLists.txt b/CMakeLists.txt index b0544c90c..474bca150 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -52,6 +52,7 @@ endif() set(DRIVER_PACKAGE_NAME "scap") include(CheckSymbolExists) + check_symbol_exists(strlcpy "string.h" HAVE_STRLCPY) if(HAVE_STRLCPY) message(STATUS "Existing strlcpy found, will *not* use local definition by setting -DHAVE_STRLCPY.") diff --git a/cmake/modules/libsinsp.cmake b/cmake/modules/libsinsp.cmake index c126359d9..e4dc8d346 100644 --- a/cmake/modules/libsinsp.cmake +++ b/cmake/modules/libsinsp.cmake @@ -17,13 +17,15 @@ include(ExternalProject) include(libscap) if(NOT WIN32) include(tbb) +endif() +if(NOT WIN32 AND NOT APPLE) include(b64) include(jq) + include(curl) endif() include(jsoncpp) if(NOT MINIMAL_BUILD) include(cares) - include(curl) endif() set(LIBSINSP_INCLUDE_DIRS ${LIBSINSP_DIR}/userspace/libsinsp ${LIBSINSP_DIR}/common ${LIBSCAP_INCLUDE_DIRS} ${DRIVER_CONFIG_DIR}) @@ -34,10 +36,6 @@ endif() if(NOT WIN32) get_filename_component(TBB_ABSOLUTE_INCLUDE_DIR ${TBB_INCLUDE_DIR} ABSOLUTE) list(APPEND LIBSINSP_INCLUDE_DIRS ${TBB_ABSOLUTE_INCLUDE_DIR}) - get_filename_component(B64_ABSOLUTE_INCLUDE_DIR ${B64_INCLUDE} ABSOLUTE) - list(APPEND LIBSINSP_INCLUDE_DIRS ${B64_ABSOLUTE_INCLUDE_DIR}) - get_filename_component(JQ_ABSOLUTE_INCLUDE_DIR ${JQ_INCLUDE} ABSOLUTE) - list(APPEND LIBSINSP_INCLUDE_DIRS ${JQ_ABSOLUTE_INCLUDE_DIR}) endif() get_filename_component(JSONCPP_ABSOLUTE_INCLUDE_DIR ${JSONCPP_INCLUDE} ABSOLUTE) @@ -45,10 +43,18 @@ list(APPEND LIBSINSP_INCLUDE_DIRS ${JSONCPP_ABSOLUTE_INCLUDE_DIR}) if(NOT MINIMAL_BUILD) get_filename_component(CARES_ABSOLUTE_INCLUDE_DIR ${CARES_INCLUDE} ABSOLUTE) list(APPEND LIBSINSP_INCLUDE_DIRS ${CARES_ABSOLUTE_INCLUDE_DIR}) +endif() + +if(NOT WIN32 AND NOT APPLE) + get_filename_component(B64_ABSOLUTE_INCLUDE_DIR ${B64_INCLUDE} ABSOLUTE) + list(APPEND LIBSINSP_INCLUDE_DIRS ${B64_ABSOLUTE_INCLUDE_DIR}) + get_filename_component(JQ_ABSOLUTE_INCLUDE_DIR ${JQ_INCLUDE} ABSOLUTE) + list(APPEND LIBSINSP_INCLUDE_DIRS ${JQ_ABSOLUTE_INCLUDE_DIR}) get_filename_component(CURL_ABSOLUTE_INCLUDE_DIR ${CURL_INCLUDE_DIR} ABSOLUTE) list(APPEND LIBSINSP_INCLUDE_DIRS ${CURL_ABSOLUTE_INCLUDE_DIR}) endif() + add_subdirectory(${LIBSINSP_DIR}/userspace/libsinsp ${CMAKE_BINARY_DIR}/libsinsp) endif() diff --git a/cmake/modules/luajit.cmake b/cmake/modules/luajit.cmake index 6cbaefc51..382dd1d7e 100644 --- a/cmake/modules/luajit.cmake +++ b/cmake/modules/luajit.cmake @@ -53,6 +53,16 @@ else() BUILD_IN_SOURCE 1 BUILD_BYPRODUCTS ${LUAJIT_LIB} INSTALL_COMMAND "") + elseif(APPLE) + ExternalProject_Add(luajit + PREFIX "${PROJECT_BINARY_DIR}/luajit-prefix" + URL "https://github.com/LuaJIT/LuaJIT/archive/v2.1.0-beta3.tar.gz" + URL_HASH "SHA256=409f7fe570d3c16558e594421c47bdd130238323c9d6fd6c83dedd2aaeb082a8" + CONFIGURE_COMMAND "" + BUILD_COMMAND make MACOSX_DEPLOYMENT_TARGET=10.14 + BUILD_IN_SOURCE 1 + BUILD_BYPRODUCTS ${LUAJIT_LIB} + INSTALL_COMMAND "") else() ExternalProject_Add(luajit PREFIX "${PROJECT_BINARY_DIR}/luajit-prefix" diff --git a/userspace/common/strlcpy.h b/userspace/common/strlcpy.h index 8a15d4051..6d71cd79c 100644 --- a/userspace/common/strlcpy.h +++ b/userspace/common/strlcpy.h @@ -26,6 +26,7 @@ limitations under the License. */ #ifndef HAVE_STRLCPY +#ifndef __APPLE__ static inline size_t strlcpy(char *dst, const char *src, size_t size) { size_t srcsize = strlen(src); if (size == 0) { @@ -44,3 +45,4 @@ static inline size_t strlcpy(char *dst, const char *src, size_t size) { return srcsize; } #endif +#endif \ No newline at end of file From 597cb94449b5b48a72c596b64262cd7a239eb094 Mon Sep 17 00:00:00 2001 From: Loris Degioanni Date: Tue, 23 Nov 2021 17:31:50 -0800 Subject: [PATCH 135/148] when calling sinsp::get_thread_ref(), covert the return value from a shared_ptr to a sinsp_threadinfo* using the shared_ptr get() method. This avoids strange behaviors with some compilers. Signed-off-by: Loris Degioanni --- userspace/libsinsp/event.cpp | 6 +++--- userspace/libsinsp/filterchecks.cpp | 10 +++++----- userspace/libsinsp/parsers.cpp | 14 +++++++------- userspace/libsinsp/sinsp.cpp | 2 +- userspace/libsinsp/threadinfo.cpp | 6 +++--- 5 files changed, 19 insertions(+), 19 deletions(-) diff --git a/userspace/libsinsp/event.cpp b/userspace/libsinsp/event.cpp index 9c95a36c8..ff52c81d1 100644 --- a/userspace/libsinsp/event.cpp +++ b/userspace/libsinsp/event.cpp @@ -138,7 +138,7 @@ sinsp_threadinfo* sinsp_evt::get_thread_info(bool query_os_if_not_found) return m_tinfo; } - return &*m_inspector->get_thread_ref(m_pevt->tid, query_os_if_not_found, false); + return m_inspector->get_thread_ref(m_pevt->tid, query_os_if_not_found, false).get(); } int64_t sinsp_evt::get_fd_num() @@ -834,7 +834,7 @@ Json::Value sinsp_evt::get_param_as_json(uint32_t id, OUT const char** resolved_ ASSERT(payload_len == sizeof(int64_t)); ret = (Json::Value::UInt64)*(int64_t *)payload; - sinsp_threadinfo* atinfo = &*m_inspector->get_thread_ref(*(int64_t *)payload, false, true); + sinsp_threadinfo* atinfo = m_inspector->get_thread_ref(*(int64_t *)payload, false, true).get(); if(atinfo != NULL) { string& tcomm = atinfo->m_comm; @@ -1553,7 +1553,7 @@ const char* sinsp_evt::get_param_as_str(uint32_t id, OUT const char** resolved_s "%" PRId64, *(int64_t *)payload); - sinsp_threadinfo* atinfo = &*m_inspector->get_thread_ref(*(int64_t *)payload, false, true); + sinsp_threadinfo* atinfo = m_inspector->get_thread_ref(*(int64_t *)payload, false, true).get(); if(atinfo != NULL) { string& tcomm = atinfo->m_comm; diff --git a/userspace/libsinsp/filterchecks.cpp b/userspace/libsinsp/filterchecks.cpp index 40c6de635..780b9edff 100644 --- a/userspace/libsinsp/filterchecks.cpp +++ b/userspace/libsinsp/filterchecks.cpp @@ -2125,7 +2125,7 @@ uint8_t* sinsp_filter_check_thread::extract(sinsp_evt *evt, OUT uint32_t* len, b // Relying on the convention that a session id is the process id of the session leader // sinsp_threadinfo* sinfo = - &*m_inspector->get_thread_ref(tinfo->m_sid, false, true); + m_inspector->get_thread_ref(tinfo->m_sid, false, true).get(); if(sinfo != NULL) { @@ -2312,7 +2312,7 @@ uint8_t* sinsp_filter_check_thread::extract(sinsp_evt *evt, OUT uint32_t* len, b case TYPE_PNAME: { sinsp_threadinfo* ptinfo = - &*m_inspector->get_thread_ref(tinfo->m_ptid, false, true); + m_inspector->get_thread_ref(tinfo->m_ptid, false, true).get(); if(ptinfo != NULL) { @@ -2327,7 +2327,7 @@ uint8_t* sinsp_filter_check_thread::extract(sinsp_evt *evt, OUT uint32_t* len, b case TYPE_PCMDLINE: { sinsp_threadinfo* ptinfo = - &*m_inspector->get_thread_ref(tinfo->m_ptid, false, true); + m_inspector->get_thread_ref(tinfo->m_ptid, false, true).get(); if(ptinfo != NULL) { @@ -5672,7 +5672,7 @@ inline uint8_t* sinsp_filter_check_evtin::extract_tracer(sinsp_evt *evt, sinsp_p // // If this is a *.p.* field, reject anything that doesn't come from the same process // - sinsp_threadinfo* tinfo = &*m_inspector->get_thread_ref(pae->m_tid); + sinsp_threadinfo* tinfo = m_inspector->get_thread_ref(pae->m_tid).get(); if(tinfo) { @@ -5693,7 +5693,7 @@ inline uint8_t* sinsp_filter_check_evtin::extract_tracer(sinsp_evt *evt, sinsp_p // // If this is a *.p.* field, reject anything that doesn't share the same parent // - sinsp_threadinfo* tinfo = &*m_inspector->get_thread_ref(pae->m_tid); + sinsp_threadinfo* tinfo = m_inspector->get_thread_ref(pae->m_tid).get(); if(tinfo) { diff --git a/userspace/libsinsp/parsers.cpp b/userspace/libsinsp/parsers.cpp index 64e288fdf..26c79f143 100644 --- a/userspace/libsinsp/parsers.cpp +++ b/userspace/libsinsp/parsers.cpp @@ -575,7 +575,7 @@ bool sinsp_parser::reset(sinsp_evt *evt) { if(etype == PPME_PROCINFO_E) { - evt->m_tinfo = &*m_inspector->get_thread_ref(evt->m_pevt->tid, false, false); + evt->m_tinfo = m_inspector->get_thread_ref(evt->m_pevt->tid, false, false).get(); } else { @@ -620,7 +620,7 @@ bool sinsp_parser::reset(sinsp_evt *evt) } else { - evt->m_tinfo = &*m_inspector->get_thread_ref(evt->m_pevt->tid, query_os, false); + evt->m_tinfo = m_inspector->get_thread_ref(evt->m_pevt->tid, query_os, false).get(); } if(etype == PPME_SCHEDSWITCH_6_E) @@ -1107,7 +1107,7 @@ void sinsp_parser::parse_clone_exit(sinsp_evt *evt) // // Lookup the thread that called clone() so we can copy its information // - sinsp_threadinfo* ptinfo = &*m_inspector->get_thread_ref(tid, true, true); + sinsp_threadinfo* ptinfo = m_inspector->get_thread_ref(tid, true, true).get(); if(NULL == ptinfo) { // @@ -1126,7 +1126,7 @@ void sinsp_parser::parse_clone_exit(sinsp_evt *evt) // // See if the child is already there // - sinsp_threadinfo* child = &*m_inspector->get_thread_ref(childtid, false, true); + sinsp_threadinfo* child = m_inspector->get_thread_ref(childtid, false, true).get(); if(NULL != child) { // @@ -1200,8 +1200,8 @@ void sinsp_parser::parse_clone_exit(sinsp_evt *evt) m_inspector->remove_thread(tid, true); tid_collision = true; - ptinfo = &*m_inspector->get_thread_ref(tid, - true, true); + ptinfo = m_inspector->get_thread_ref(tid, + true, true).get(); if(ptinfo == NULL) { @@ -4340,7 +4340,7 @@ void sinsp_parser::parse_prlimit_exit(sinsp_evt *evt) tid = evt->get_tid(); } - sinsp_threadinfo* ptinfo = &*m_inspector->get_thread_ref(tid, true, true); + sinsp_threadinfo* ptinfo = m_inspector->get_thread_ref(tid, true, true).get(); if(ptinfo == NULL) { ASSERT(false); diff --git a/userspace/libsinsp/sinsp.cpp b/userspace/libsinsp/sinsp.cpp index 1bb17ca1e..e0167c381 100644 --- a/userspace/libsinsp/sinsp.cpp +++ b/userspace/libsinsp/sinsp.cpp @@ -1288,7 +1288,7 @@ int32_t sinsp::next(OUT sinsp_evt **puevt) if(nfdr != 0) { - sinsp_threadinfo* ptinfo = &*get_thread_ref(m_tid_of_fd_to_remove, true, true); + sinsp_threadinfo* ptinfo = get_thread_ref(m_tid_of_fd_to_remove, true, true).get(); if(!ptinfo) { ASSERT(false); diff --git a/userspace/libsinsp/threadinfo.cpp b/userspace/libsinsp/threadinfo.cpp index 77d015267..0c004afdc 100644 --- a/userspace/libsinsp/threadinfo.cpp +++ b/userspace/libsinsp/threadinfo.cpp @@ -665,7 +665,7 @@ void sinsp_threadinfo::set_cgroups(const char* cgroups, size_t len) sinsp_threadinfo* sinsp_threadinfo::get_parent_thread() { - return &*m_inspector->get_thread_ref(m_ptid, false, true); + return m_inspector->get_thread_ref(m_ptid, false, true).get(); } sinsp_fdinfo_t* sinsp_threadinfo::add_fd(int64_t fd, sinsp_fdinfo_t *fdinfo) @@ -1243,7 +1243,7 @@ void sinsp_thread_manager::increment_mainthread_childcount(sinsp_threadinfo* thr // ASSERT(threadinfo->m_pid != threadinfo->m_tid); - sinsp_threadinfo* main_thread = &*m_inspector->get_thread_ref(threadinfo->m_pid, true, true); + sinsp_threadinfo* main_thread = m_inspector->get_thread_ref(threadinfo->m_pid, true, true).get(); if(main_thread) { ++main_thread->m_nchilds; @@ -1318,7 +1318,7 @@ void sinsp_thread_manager::remove_thread(int64_t tid, bool force) if(tinfo->m_flags & PPM_CL_CLONE_THREAD) { ASSERT(tinfo->m_pid != tinfo->m_tid); - sinsp_threadinfo* main_thread = &*m_inspector->get_thread_ref(tinfo->m_pid, false, true); + sinsp_threadinfo* main_thread = m_inspector->get_thread_ref(tinfo->m_pid, false, true).get(); if(main_thread) { if(main_thread->m_nchilds > 0) From 41ae5b776bf51e4fa50f14e694380897d912c185 Mon Sep 17 00:00:00 2001 From: Loris Degioanni Date: Fri, 26 Nov 2021 09:25:41 -0800 Subject: [PATCH 136/148] make sure curl is not used in the minimal build (it also needs to be excluded under Windows and Mac Signed-off-by: Loris Degioanni --- cmake/modules/libsinsp.cmake | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/cmake/modules/libsinsp.cmake b/cmake/modules/libsinsp.cmake index e4dc8d346..f09070d57 100644 --- a/cmake/modules/libsinsp.cmake +++ b/cmake/modules/libsinsp.cmake @@ -21,6 +21,8 @@ endif() if(NOT WIN32 AND NOT APPLE) include(b64) include(jq) +endif() +if(NOT WIN32 AND NOT APPLE AND NOT MINIMAL_BUILD) include(curl) endif() include(jsoncpp) @@ -54,7 +56,6 @@ if(NOT WIN32 AND NOT APPLE) list(APPEND LIBSINSP_INCLUDE_DIRS ${CURL_ABSOLUTE_INCLUDE_DIR}) endif() - add_subdirectory(${LIBSINSP_DIR}/userspace/libsinsp ${CMAKE_BINARY_DIR}/libsinsp) endif() From c6cbde657e40d81f0ca9d3f4468aa2bd7db26022 Mon Sep 17 00:00:00 2001 From: Loris Degioanni Date: Fri, 26 Nov 2021 09:42:39 -0800 Subject: [PATCH 137/148] don't use the OS strlcpy on Mac Signed-off-by: Loris Degioanni --- CMakeLists.txt | 2 +- userspace/common/strlcpy.h | 2 -- 2 files changed, 1 insertion(+), 3 deletions(-) diff --git a/CMakeLists.txt b/CMakeLists.txt index 474bca150..78e7040b5 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -54,7 +54,7 @@ set(DRIVER_PACKAGE_NAME "scap") include(CheckSymbolExists) check_symbol_exists(strlcpy "string.h" HAVE_STRLCPY) -if(HAVE_STRLCPY) +if(HAVE_STRLCPY AND NOT APPLE) message(STATUS "Existing strlcpy found, will *not* use local definition by setting -DHAVE_STRLCPY.") add_definitions(-DHAVE_STRLCPY) else() diff --git a/userspace/common/strlcpy.h b/userspace/common/strlcpy.h index 6d71cd79c..091710b71 100644 --- a/userspace/common/strlcpy.h +++ b/userspace/common/strlcpy.h @@ -26,7 +26,6 @@ limitations under the License. */ #ifndef HAVE_STRLCPY -#ifndef __APPLE__ static inline size_t strlcpy(char *dst, const char *src, size_t size) { size_t srcsize = strlen(src); if (size == 0) { @@ -44,5 +43,4 @@ static inline size_t strlcpy(char *dst, const char *src, size_t size) { return srcsize; } -#endif #endif \ No newline at end of file From c432afad7b35b70356a5c8e27e99df26f8a64dd4 Mon Sep 17 00:00:00 2001 From: Loris Degioanni Date: Fri, 26 Nov 2021 09:44:27 -0800 Subject: [PATCH 138/148] cleanups Signed-off-by: Loris Degioanni --- CMakeLists.txt | 1 - 1 file changed, 1 deletion(-) diff --git a/CMakeLists.txt b/CMakeLists.txt index 78e7040b5..c2866980f 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -52,7 +52,6 @@ endif() set(DRIVER_PACKAGE_NAME "scap") include(CheckSymbolExists) - check_symbol_exists(strlcpy "string.h" HAVE_STRLCPY) if(HAVE_STRLCPY AND NOT APPLE) message(STATUS "Existing strlcpy found, will *not* use local definition by setting -DHAVE_STRLCPY.") From 14e496c9b4f8731e699ac3ce439d2faa632e34d4 Mon Sep 17 00:00:00 2001 From: Loris Degioanni Date: Fri, 26 Nov 2021 14:31:11 -0800 Subject: [PATCH 139/148] apple check not needed when detecting strlcpy in the makefile Signed-off-by: Loris Degioanni --- CMakeLists.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CMakeLists.txt b/CMakeLists.txt index c2866980f..b0544c90c 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -53,7 +53,7 @@ set(DRIVER_PACKAGE_NAME "scap") include(CheckSymbolExists) check_symbol_exists(strlcpy "string.h" HAVE_STRLCPY) -if(HAVE_STRLCPY AND NOT APPLE) +if(HAVE_STRLCPY) message(STATUS "Existing strlcpy found, will *not* use local definition by setting -DHAVE_STRLCPY.") add_definitions(-DHAVE_STRLCPY) else() From b0a30dc95709405e5da817f5e800714dca88563e Mon Sep 17 00:00:00 2001 From: Loris Degioanni Date: Fri, 26 Nov 2021 14:36:36 -0800 Subject: [PATCH 140/148] cleanups Signed-off-by: Loris Degioanni --- userspace/common/strlcpy.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/userspace/common/strlcpy.h b/userspace/common/strlcpy.h index 091710b71..8a15d4051 100644 --- a/userspace/common/strlcpy.h +++ b/userspace/common/strlcpy.h @@ -43,4 +43,4 @@ static inline size_t strlcpy(char *dst, const char *src, size_t size) { return srcsize; } -#endif \ No newline at end of file +#endif From 2160111cd088aea9ae2235d3385ecb0b1ab6623c Mon Sep 17 00:00:00 2001 From: Loris Degioanni Date: Fri, 26 Nov 2021 14:40:12 -0800 Subject: [PATCH 141/148] make sure curl is actually not include for minimal build Signed-off-by: Loris Degioanni --- cmake/modules/libsinsp.cmake | 3 +++ 1 file changed, 3 insertions(+) diff --git a/cmake/modules/libsinsp.cmake b/cmake/modules/libsinsp.cmake index f09070d57..802726ce5 100644 --- a/cmake/modules/libsinsp.cmake +++ b/cmake/modules/libsinsp.cmake @@ -52,6 +52,9 @@ if(NOT WIN32 AND NOT APPLE) list(APPEND LIBSINSP_INCLUDE_DIRS ${B64_ABSOLUTE_INCLUDE_DIR}) get_filename_component(JQ_ABSOLUTE_INCLUDE_DIR ${JQ_INCLUDE} ABSOLUTE) list(APPEND LIBSINSP_INCLUDE_DIRS ${JQ_ABSOLUTE_INCLUDE_DIR}) +endif() + +if(NOT WIN32 AND NOT APPLE AND NOT MINIMAL_BUILD) get_filename_component(CURL_ABSOLUTE_INCLUDE_DIR ${CURL_INCLUDE_DIR} ABSOLUTE) list(APPEND LIBSINSP_INCLUDE_DIRS ${CURL_ABSOLUTE_INCLUDE_DIR}) endif() From 0fbdd13947706e2e4b7f5c04ffb52dbc6d1d16e0 Mon Sep 17 00:00:00 2001 From: Federico Di Pierro Date: Wed, 8 Sep 2021 10:21:23 +0200 Subject: [PATCH 142/148] fix(userspace/libscap): improve users loading code robustness: we need 2 passes to load users: a first scan to count number of users, and a second scan to actually fill our data structures. Previously, if any user was added in between these 2 steps, we would've segfaulted. Now we will "lose" the new user but we won't crash. While this may seem an unnecessary check, there may be cases (even if never reported) where eg: falco is run as a systemd service and starts very soon during boot up, and some other systemd unit running in parallel is creating some users. The same fix applies to groups loading code too. Signed-off-by: Federico Di Pierro --- userspace/libscap/scap_userlist.c | 70 +++++++++++++++++++++---------- 1 file changed, 48 insertions(+), 22 deletions(-) diff --git a/userspace/libscap/scap_userlist.c b/userspace/libscap/scap_userlist.c index 5fe3521d6..d09ce743b 100644 --- a/userspace/libscap/scap_userlist.c +++ b/userspace/libscap/scap_userlist.c @@ -31,8 +31,8 @@ limitations under the License. // int32_t scap_create_userlist(scap_t* handle) { - uint32_t usercnt; - uint32_t grpcnt; + uint32_t usercnt, useridx; + uint32_t grpcnt, grpidx; struct passwd *p; struct group *g; @@ -97,74 +97,100 @@ int32_t scap_create_userlist(scap_t* handle) setpwent(); p = getpwent(); - for(usercnt = 0; p; p = getpwent(), usercnt++) + for(useridx = 0; useridx < usercnt && p; p = getpwent(), useridx++) { - handle->m_userlist->users[usercnt].uid = p->pw_uid; - handle->m_userlist->users[usercnt].gid = p->pw_gid; + scap_userinfo *user = &handle->m_userlist->users[useridx]; + user->uid = p->pw_uid; + user->gid = p->pw_gid; if(p->pw_name) { - strlcpy(handle->m_userlist->users[usercnt].name, p->pw_name, sizeof(handle->m_userlist->users[usercnt].name)); + strlcpy(user->name, p->pw_name, sizeof(user->name)); } else { - *handle->m_userlist->users[usercnt].name = '\0'; + *user->name = '\0'; } if(p->pw_dir) { - strlcpy(handle->m_userlist->users[usercnt].homedir, p->pw_dir, sizeof(handle->m_userlist->users[usercnt].homedir)); + strlcpy(user->homedir, p->pw_dir, sizeof(user->homedir)); } else { - *handle->m_userlist->users[usercnt].homedir = '\0'; + *user->homedir = '\0'; } if(p->pw_shell) { - strlcpy(handle->m_userlist->users[usercnt].shell, p->pw_shell, sizeof(handle->m_userlist->users[usercnt].shell)); + strlcpy(user->shell, p->pw_shell, sizeof(user->shell)); } else { - *handle->m_userlist->users[usercnt].shell = '\0'; + *user->shell = '\0'; } handle->m_userlist->totsavelen += sizeof(uint8_t) + // type - sizeof(uint32_t) + // uid - sizeof(uint32_t) + // gid - strlen(handle->m_userlist->users[usercnt].name) + 2 + - strlen(handle->m_userlist->users[usercnt].homedir) + 2 + - strlen(handle->m_userlist->users[usercnt].shell) + 2; + sizeof(user->uid) + + sizeof(user->gid) + + strlen(user->name) + 2 + + strlen(user->homedir) + 2 + + strlen(user->shell) + 2; } endpwent(); + /* + * Check that no user was removed between the 2 iterations of users; + * we don't really care if any user was added instead; + * we will just miss the last user returned in that case. + */ + if (useridx < usercnt) { + // Any user was removed while we were cycling + handle->m_userlist->nusers = useridx; + // we are reducing the allocated area; no need to check that realloc is fine + handle->m_userlist->users = realloc(handle->m_userlist->users, useridx * sizeof(scap_userinfo)); + } + // groups setgrent(); g = getgrent(); - for(grpcnt = 0; g; g = getgrent(), grpcnt++) + for(grpidx = 0; grpidx < grpcnt && g; g = getgrent(), grpidx++) { - handle->m_userlist->groups[grpcnt].gid = g->gr_gid; + scap_groupinfo *group = &handle->m_userlist->groups[grpidx]; + group->gid = g->gr_gid; if(g->gr_name) { - strlcpy(handle->m_userlist->groups[grpcnt].name, g->gr_name, sizeof(handle->m_userlist->groups[grpcnt].name)); + strlcpy(group->name, g->gr_name, sizeof(group->name)); } else { - *handle->m_userlist->groups[grpcnt].name = '\0'; + *group->name = '\0'; } handle->m_userlist->totsavelen += sizeof(uint8_t) + // type - sizeof(uint32_t) + // gid - strlen(handle->m_userlist->groups[grpcnt].name) + 2; + sizeof(group->gid) + + strlen(group->name) + 2; } endgrent(); + /* + * Check that no group was removed between the 2 iterations of groups; + * we don't really care if any group was added instead; + * we will just miss the last group returned in that case. + */ + if (grpidx < grpcnt) { + // Any group was removed while we were cycling + handle->m_userlist->ngroups = grpidx; + // we are reducing the allocated area; no need to check that realloc is fine + handle->m_userlist->groups = realloc(handle->m_userlist->groups, grpidx * sizeof(scap_groupinfo)); + } + return SCAP_SUCCESS; } #else // HAS_CAPTURE From 8a1db1f21d1067121421507e50873a2d1918d755 Mon Sep 17 00:00:00 2001 From: Federico Di Pierro Date: Wed, 8 Sep 2021 10:26:24 +0200 Subject: [PATCH 143/148] repo(.gitignore): updated gitignore adding .idea folder (for CLion users) Signed-off-by: Federico Di Pierro --- .gitignore | 1 + 1 file changed, 1 insertion(+) diff --git a/.gitignore b/.gitignore index 3b9a3efce..c964edc86 100644 --- a/.gitignore +++ b/.gitignore @@ -23,3 +23,4 @@ driver/driver_config.h .vscode .cache.mk build* +.idea From c954cc3057b96f5546143082b3796133790d9d8d Mon Sep 17 00:00:00 2001 From: Federico Di Pierro Date: Wed, 8 Sep 2021 10:27:45 +0200 Subject: [PATCH 144/148] refactor(userspace/libsinsp): added a sinsp::get_group() api, similar to sinsp::get_user(), and use it where needed. Moreover, drop an useless assert() in sinsp_filter_check_user::extract(). Signed-off-by: Federico Di Pierro --- userspace/libsinsp/event.cpp | 10 ++++------ userspace/libsinsp/filterchecks.cpp | 19 +------------------ userspace/libsinsp/sinsp.cpp | 19 +++++++++++++++++-- userspace/libsinsp/sinsp.h | 14 +++++++++++++- 4 files changed, 35 insertions(+), 27 deletions(-) diff --git a/userspace/libsinsp/event.cpp b/userspace/libsinsp/event.cpp index ff52c81d1..be6db5668 100644 --- a/userspace/libsinsp/event.cpp +++ b/userspace/libsinsp/event.cpp @@ -2162,10 +2162,9 @@ const char* sinsp_evt::get_param_as_str(uint32_t id, OUT const char** resolved_s snprintf(&m_paramstr_storage[0], m_paramstr_storage.size(), "%d", val); - auto find_it = m_inspector->get_userlist()->find(val); - if (find_it != m_inspector->get_userlist()->end()) + auto user_info = m_inspector->get_user(val); + if (user_info != NULL) { - scap_userinfo* user_info = find_it->second; strcpy_sanitized(&m_resolved_paramstr_storage[0], user_info->name, (uint32_t)m_resolved_paramstr_storage.size()); } @@ -2195,10 +2194,9 @@ const char* sinsp_evt::get_param_as_str(uint32_t id, OUT const char** resolved_s snprintf(&m_paramstr_storage[0], m_paramstr_storage.size(), "%d", val); - auto find_it = m_inspector->get_grouplist()->find(val); - if (find_it != m_inspector->get_grouplist()->end()) + auto group_info = m_inspector->get_group(val); + if (group_info != NULL) { - scap_groupinfo* group_info = find_it->second; strcpy_sanitized(&m_resolved_paramstr_storage[0], group_info->name, (uint32_t)m_resolved_paramstr_storage.size()); } diff --git a/userspace/libsinsp/filterchecks.cpp b/userspace/libsinsp/filterchecks.cpp index 780b9edff..3ee88308e 100644 --- a/userspace/libsinsp/filterchecks.cpp +++ b/userspace/libsinsp/filterchecks.cpp @@ -4755,7 +4755,6 @@ uint8_t* sinsp_filter_check_user::extract(sinsp_evt *evt, OUT uint32_t* len, boo { ASSERT(m_inspector != NULL); uinfo = m_inspector->get_user(tinfo->m_uid); - ASSERT(uinfo != NULL); if(uinfo == NULL) { return NULL; @@ -4832,23 +4831,7 @@ uint8_t* sinsp_filter_check_group::extract(sinsp_evt *evt, OUT uint32_t* len, bo unordered_map::iterator it; ASSERT(m_inspector != NULL); - unordered_map* grouplist = - (unordered_map*)m_inspector->get_grouplist(); - ASSERT(grouplist->size() != 0); - - if(tinfo->m_gid == 0xffffffff) - { - return NULL; - } - - it = grouplist->find(tinfo->m_gid); - if(it == grouplist->end()) - { - ASSERT(false); - return NULL; - } - - scap_groupinfo* ginfo = it->second; + auto ginfo = m_inspector->get_group(tinfo->m_gid); ASSERT(ginfo != NULL); RETURN_EXTRACT_CSTR(ginfo->name); diff --git a/userspace/libsinsp/sinsp.cpp b/userspace/libsinsp/sinsp.cpp index e0167c381..945441fb7 100644 --- a/userspace/libsinsp/sinsp.cpp +++ b/userspace/libsinsp/sinsp.cpp @@ -1799,13 +1799,12 @@ const unordered_map* sinsp::get_userlist() scap_userinfo* sinsp::get_user(uint32_t uid) { - unordered_map::const_iterator it; if(uid == 0xffffffff) { return NULL; } - it = m_userlist.find(uid); + auto it = m_userlist.find(uid); if(it == m_userlist.end()) { return NULL; @@ -1819,6 +1818,22 @@ const unordered_map* sinsp::get_grouplist() return &m_grouplist; } +scap_groupinfo* sinsp::get_group(uint32_t gid) +{ + if(gid == 0xffffffff) + { + return NULL; + } + + auto it = m_grouplist.find(gid); + if(it == m_grouplist.end()) + { + return NULL; + } + + return it->second; +} + #ifdef HAS_FILTERING void sinsp::get_filtercheck_fields_info(OUT vector& list) { diff --git a/userspace/libsinsp/sinsp.h b/userspace/libsinsp/sinsp.h index 4928f7afa..a146581bb 100644 --- a/userspace/libsinsp/sinsp.h +++ b/userspace/libsinsp/sinsp.h @@ -491,7 +491,7 @@ class SINSP_PUBLIC sinsp : public capture_stats_source, public wmi_handle_source threadinfo_map_t::ptr_t get_thread_ref(int64_t tid, bool query_os_if_not_found = false, bool lookup_only = true, bool main_thread = false); /*! - \brief Return the table with all the machine users. + \brief Return the table with all the machine users. \return a hash table with the user ID (UID) as the key and the user information as the data. @@ -526,6 +526,18 @@ class SINSP_PUBLIC sinsp : public capture_stats_source, public wmi_handle_source */ const unordered_map* get_grouplist(); + /*! + \brief Lookup for group in the group table. + + \return the \ref scap_groupinfo object containing full group information, + if group not found, returns NULL. + + \note this call works with file captures as well, because the group + table is stored in the trace files. In that case, the returned + group list is the one of the machine where the capture happened. + */ + scap_groupinfo* get_group(uint32_t gid); + /*! \brief Fill the given structure with statistics about the currently open capture. From f522165c917d989288320a64ecf0cbe45ab609df Mon Sep 17 00:00:00 2001 From: Federico Di Pierro Date: Wed, 8 Sep 2021 12:16:45 +0200 Subject: [PATCH 145/148] fix(userspace/libsinsp): added back assert() that was removed last commit, plus add another assert() still in sinsp_filter_check_user::extract() after get_user() call. On a second thought, it is better to leave them here as they alert that something is weird, and they only work for debug builds. Signed-off-by: Federico Di Pierro --- userspace/libsinsp/filterchecks.cpp | 2 ++ 1 file changed, 2 insertions(+) diff --git a/userspace/libsinsp/filterchecks.cpp b/userspace/libsinsp/filterchecks.cpp index 3ee88308e..93c44a3a7 100644 --- a/userspace/libsinsp/filterchecks.cpp +++ b/userspace/libsinsp/filterchecks.cpp @@ -4755,6 +4755,7 @@ uint8_t* sinsp_filter_check_user::extract(sinsp_evt *evt, OUT uint32_t* len, boo { ASSERT(m_inspector != NULL); uinfo = m_inspector->get_user(tinfo->m_uid); + ASSERT(uinfo != NULL); if(uinfo == NULL) { return NULL; @@ -4776,6 +4777,7 @@ uint8_t* sinsp_filter_check_user::extract(sinsp_evt *evt, OUT uint32_t* len, boo case TYPE_LOGINNAME: ASSERT(m_inspector != NULL); uinfo = m_inspector->get_user(tinfo->m_loginuid); + ASSERT(uinfo != NULL); if(uinfo == NULL) { return NULL; From 3ee8453b2d83de0b0f4f87d6177a01d052dbf206 Mon Sep 17 00:00:00 2001 From: Federico Di Pierro Date: Mon, 29 Nov 2021 14:37:52 +0100 Subject: [PATCH 146/148] update(userspace/libscap, userspace/libsinsp): addressed review comments. Avoid 2-pass scan of users and groups. Fixed up some small issues. Signed-off-by: Federico Di Pierro Co-authored-by: Grzegorz Nosek --- .gitignore | 1 - userspace/libscap/scap_userlist.c | 97 +++++++++++++++-------------- userspace/libsinsp/filterchecks.cpp | 7 ++- userspace/libsinsp/sinsp.h | 2 +- 4 files changed, 56 insertions(+), 51 deletions(-) diff --git a/.gitignore b/.gitignore index c964edc86..3b9a3efce 100644 --- a/.gitignore +++ b/.gitignore @@ -23,4 +23,3 @@ driver/driver_config.h .vscode .cache.mk build* -.idea diff --git a/userspace/libscap/scap_userlist.c b/userspace/libscap/scap_userlist.c index d09ce743b..7fba2b6b1 100644 --- a/userspace/libscap/scap_userlist.c +++ b/userspace/libscap/scap_userlist.c @@ -46,19 +46,6 @@ int32_t scap_create_userlist(scap_t* handle) handle->m_userlist = NULL; } - // - // First pass: count the number of users and the number of groups - // - setpwent(); - p = getpwent(); - for(usercnt = 0; p; p = getpwent(), usercnt++); - endpwent(); - - setgrent(); - g = getgrent(); - for(grpcnt = 0; g; g = getgrent(), grpcnt++); - endgrent(); - // // Memory allocations // @@ -69,9 +56,8 @@ int32_t scap_create_userlist(scap_t* handle) return SCAP_FAILURE; } - handle->m_userlist->nusers = usercnt; - handle->m_userlist->ngroups = grpcnt; handle->m_userlist->totsavelen = 0; + usercnt = 32; // initial user count; will be realloc'd if needed handle->m_userlist->users = (scap_userinfo*)malloc(usercnt * sizeof(scap_userinfo)); if(handle->m_userlist->users == NULL) { @@ -80,6 +66,7 @@ int32_t scap_create_userlist(scap_t* handle) return SCAP_FAILURE; } + grpcnt = 32; // initial group count; will be realloc'd if needed handle->m_userlist->groups = (scap_groupinfo*)malloc(grpcnt * sizeof(scap_groupinfo)); if(handle->m_userlist->groups == NULL) { @@ -89,16 +76,30 @@ int32_t scap_create_userlist(scap_t* handle) return SCAP_FAILURE; } - // - // Second pass: copy the data - // - - //users + // users setpwent(); p = getpwent(); - for(useridx = 0; useridx < usercnt && p; p = getpwent(), useridx++) + for(useridx = 0; p; p = getpwent(), useridx++) { + if (useridx == usercnt) + { + usercnt<<=1; + void *tmp = realloc(handle->m_userlist->users, usercnt * sizeof(scap_userinfo)); + if (tmp) + { + handle->m_userlist->users = tmp; + } + else + { + snprintf(handle->m_lasterr, SCAP_LASTERR_SIZE, "userlist allocation failed(2)"); + free(handle->m_userlist->users); + free(handle->m_userlist->groups); + free(handle->m_userlist); + return SCAP_FAILURE; + } + } + scap_userinfo *user = &handle->m_userlist->users[useridx]; user->uid = p->pw_uid; user->gid = p->pw_gid; @@ -132,24 +133,18 @@ int32_t scap_create_userlist(scap_t* handle) handle->m_userlist->totsavelen += sizeof(uint8_t) + // type - sizeof(user->uid) + - sizeof(user->gid) + + sizeof(uint32_t) + // uid + sizeof(uint32_t) + // gid strlen(user->name) + 2 + strlen(user->homedir) + 2 + strlen(user->shell) + 2; } endpwent(); - - /* - * Check that no user was removed between the 2 iterations of users; - * we don't really care if any user was added instead; - * we will just miss the last user returned in that case. - */ - if (useridx < usercnt) { - // Any user was removed while we were cycling - handle->m_userlist->nusers = useridx; - // we are reducing the allocated area; no need to check that realloc is fine + handle->m_userlist->nusers = useridx; + if (useridx < usercnt) + { + // Reduce array size handle->m_userlist->users = realloc(handle->m_userlist->users, useridx * sizeof(scap_userinfo)); } @@ -157,8 +152,25 @@ int32_t scap_create_userlist(scap_t* handle) setgrent(); g = getgrent(); - for(grpidx = 0; grpidx < grpcnt && g; g = getgrent(), grpidx++) + for(grpidx = 0; g; g = getgrent(), grpidx++) { + if (grpidx == grpcnt) + { + grpcnt<<=1; + void *tmp = realloc(handle->m_userlist->groups, grpcnt * sizeof(scap_groupinfo)); + if (tmp) + { + handle->m_userlist->groups = tmp; + } + else + { + snprintf(handle->m_lasterr, SCAP_LASTERR_SIZE, "grouplist allocation failed(2)"); + free(handle->m_userlist->users); + free(handle->m_userlist->groups); + free(handle->m_userlist); + return SCAP_FAILURE; + } + } scap_groupinfo *group = &handle->m_userlist->groups[grpidx]; group->gid = g->gr_gid; @@ -173,24 +185,17 @@ int32_t scap_create_userlist(scap_t* handle) handle->m_userlist->totsavelen += sizeof(uint8_t) + // type - sizeof(group->gid) + + sizeof(uint32_t) + // gid strlen(group->name) + 2; } endgrent(); - - /* - * Check that no group was removed between the 2 iterations of groups; - * we don't really care if any group was added instead; - * we will just miss the last group returned in that case. - */ - if (grpidx < grpcnt) { - // Any group was removed while we were cycling - handle->m_userlist->ngroups = grpidx; - // we are reducing the allocated area; no need to check that realloc is fine + handle->m_userlist->ngroups = grpidx; + if (grpidx < grpcnt) + { + // Reduce array size handle->m_userlist->groups = realloc(handle->m_userlist->groups, grpidx * sizeof(scap_groupinfo)); } - return SCAP_SUCCESS; } #else // HAS_CAPTURE diff --git a/userspace/libsinsp/filterchecks.cpp b/userspace/libsinsp/filterchecks.cpp index 93c44a3a7..6326a3770 100644 --- a/userspace/libsinsp/filterchecks.cpp +++ b/userspace/libsinsp/filterchecks.cpp @@ -4777,7 +4777,6 @@ uint8_t* sinsp_filter_check_user::extract(sinsp_evt *evt, OUT uint32_t* len, boo case TYPE_LOGINNAME: ASSERT(m_inspector != NULL); uinfo = m_inspector->get_user(tinfo->m_loginuid); - ASSERT(uinfo != NULL); if(uinfo == NULL) { return NULL; @@ -4834,8 +4833,10 @@ uint8_t* sinsp_filter_check_group::extract(sinsp_evt *evt, OUT uint32_t* len, bo ASSERT(m_inspector != NULL); auto ginfo = m_inspector->get_group(tinfo->m_gid); - ASSERT(ginfo != NULL); - + if (ginfo == NULL) + { + return NULL; + } RETURN_EXTRACT_CSTR(ginfo->name); } default: diff --git a/userspace/libsinsp/sinsp.h b/userspace/libsinsp/sinsp.h index a146581bb..620bc1c87 100644 --- a/userspace/libsinsp/sinsp.h +++ b/userspace/libsinsp/sinsp.h @@ -491,7 +491,7 @@ class SINSP_PUBLIC sinsp : public capture_stats_source, public wmi_handle_source threadinfo_map_t::ptr_t get_thread_ref(int64_t tid, bool query_os_if_not_found = false, bool lookup_only = true, bool main_thread = false); /*! - \brief Return the table with all the machine users. + \brief Return the table with all the machine users. \return a hash table with the user ID (UID) as the key and the user information as the data. From 3cc178e06086c126fa539a9d82b3c616796b56a9 Mon Sep 17 00:00:00 2001 From: Grzegorz Nosek Date: Mon, 13 Dec 2021 14:13:25 +0100 Subject: [PATCH 147/148] Make the `scap-driver` component name configurable This affects downstream packaging (the cmake component name ends up in package names) so make this configurable instead of forcing `scap-driver`. Signed-off-by: Angelo Puglisi --- driver/CMakeLists.txt | 5 ++++- driver/bpf/CMakeLists.txt | 2 +- 2 files changed, 5 insertions(+), 2 deletions(-) diff --git a/driver/CMakeLists.txt b/driver/CMakeLists.txt index b446ab4d8..2d23d8058 100644 --- a/driver/CMakeLists.txt +++ b/driver/CMakeLists.txt @@ -7,6 +7,9 @@ option(BUILD_DRIVER "Build the driver on Linux" ON) option(ENABLE_DKMS "Enable DKMS on Linux" ON) +if(NOT DEFINED DRIVER_COMPONENT_NAME) + set(DRIVER_COMPONENT_NAME "scap-driver") +endif() # The driver build process is somewhat involved because we use the same # sources for building the driver locally and for shipping as a DKMS module. @@ -104,7 +107,7 @@ if(ENABLE_DKMS) ${CMAKE_CURRENT_BINARY_DIR}/src/driver_config.h ${DRIVER_SOURCES} DESTINATION "src/${DRIVER_PACKAGE_NAME}-${PROBE_VERSION}" - COMPONENT scap-driver) + COMPONENT ${DRIVER_COMPONENT_NAME}) endif() diff --git a/driver/bpf/CMakeLists.txt b/driver/bpf/CMakeLists.txt index 99321d6d0..cf19f5037 100644 --- a/driver/bpf/CMakeLists.txt +++ b/driver/bpf/CMakeLists.txt @@ -30,4 +30,4 @@ install(FILES ring_helpers.h types.h DESTINATION "src/${DRIVER_PACKAGE_NAME}-${PROBE_VERSION}/bpf" - COMPONENT scap-driver) + COMPONENT ${DRIVER_COMPONENT_NAME}) From b573c6e6203272e6738020c83cf0d09078f08eb6 Mon Sep 17 00:00:00 2001 From: Mujahid-Dandoti Date: Sun, 14 Nov 2021 21:59:44 +0530 Subject: [PATCH 148/148] Updated protobuf version for s390x Signed-off-by: Mujahid-Dandoti --- cmake/modules/protobuf.cmake | 38 +++++++++++------------------------- 1 file changed, 11 insertions(+), 27 deletions(-) diff --git a/cmake/modules/protobuf.cmake b/cmake/modules/protobuf.cmake index 623cd814d..1ed9f954b 100644 --- a/cmake/modules/protobuf.cmake +++ b/cmake/modules/protobuf.cmake @@ -24,33 +24,17 @@ else() if(NOT TARGET protobuf) message(STATUS "Using bundled protobuf in '${PROTOBUF_SRC}'") - if("${CMAKE_SYSTEM_PROCESSOR}" STREQUAL "s390x") - ExternalProject_Add(protobuf - PREFIX "${PROJECT_BINARY_DIR}/protobuf-prefix" - DEPENDS openssl zlib - URL "http://download.sysdig.com/dependencies/protobuf-cpp-3.5.0.tar.gz" - URL_MD5 "e4ba8284a407712168593e79e6555eb2" - PATCH_COMMAND wget http://download.sysdig.com/dependencies/protobuf-3.5.0-s390x.patch && patch -p1 -i protobuf-3.5.0-s390x.patch - # TODO what if using system zlib? - CONFIGURE_COMMAND /usr/bin/env CPPFLAGS=-I${ZLIB_INCLUDE} LDFLAGS=-L${ZLIB_SRC} ./configure --with-zlib --disable-shared --enable-static --prefix=${PROTOBUF_INSTALL_DIR} - COMMAND aclocal && automake - BUILD_COMMAND ${CMD_MAKE} - BUILD_IN_SOURCE 1 - BUILD_BYPRODUCTS ${PROTOC} ${PROTOBUF_INCLUDE} ${PROTOBUF_LIB} - INSTALL_COMMAND make install) - else() - ExternalProject_Add(protobuf - PREFIX "${PROJECT_BINARY_DIR}/protobuf-prefix" - DEPENDS openssl zlib - URL "https://github.com/protocolbuffers/protobuf/releases/download/v3.17.3/protobuf-cpp-3.17.3.tar.gz" - URL_HASH "SHA256=51cec99f108b83422b7af1170afd7aeb2dd77d2bcbb7b6bad1f92509e9ccf8cb" - # TODO what if using system zlib? - CONFIGURE_COMMAND /usr/bin/env CPPFLAGS=-I${ZLIB_INCLUDE} LDFLAGS=-L${ZLIB_SRC} ./configure --with-zlib --disable-shared --enable-static --prefix=${PROTOBUF_INSTALL_DIR} - BUILD_COMMAND ${CMD_MAKE} - BUILD_IN_SOURCE 1 - BUILD_BYPRODUCTS ${PROTOC} ${PROTOBUF_INCLUDE} ${PROTOBUF_LIB} - INSTALL_COMMAND make install) - endif() + ExternalProject_Add(protobuf + PREFIX "${PROJECT_BINARY_DIR}/protobuf-prefix" + DEPENDS openssl zlib + URL "https://github.com/protocolbuffers/protobuf/releases/download/v3.17.3/protobuf-cpp-3.17.3.tar.gz" + URL_HASH "SHA256=51cec99f108b83422b7af1170afd7aeb2dd77d2bcbb7b6bad1f92509e9ccf8cb" + # TODO what if using system zlib? + CONFIGURE_COMMAND /usr/bin/env CPPFLAGS=-I${ZLIB_INCLUDE} LDFLAGS=-L${ZLIB_SRC} ./configure --with-zlib --disable-shared --enable-static --prefix=${PROTOBUF_INSTALL_DIR} + BUILD_COMMAND ${CMD_MAKE} + BUILD_IN_SOURCE 1 + BUILD_BYPRODUCTS ${PROTOC} ${PROTOBUF_INCLUDE} ${PROTOBUF_LIB} + INSTALL_COMMAND make install) endif() endif()