I have basic question on ebpf behavior when multiple ebpf hooks are loaded(not using chaining) simultaneously in kernel. Are all of those hooks invoked? For example, I loaded my ebpf program which has cgroup_skb/ingress hook. The ebpf hooks functions loaded by my program are not getting invoked by kernel during packet ingress. When I list loaded programs on "Linux 5.13.0-30-generic", I see that systemd has already loaded cgroup_skb/ingress hooks by default. I do understand why these cgroup/skb hooks are loaded by systemd. My question is specifically on kernel's behavior when multiple hooks of same type(cgroup_skb/ingress) are loaded.
yes. All the hooks will be called. We need to attach it with "multi" keyword.
bpftool prog load test1.o /sys/fs/bpf/test1 type cgroup/skb
bpftool cgroup attach /sys/fs/cgroup/mygrp egress pinned /sys/fs/bpf/test1 multi
bpftool prog load test2.o /sys/fs/bpf/test2 type cgroup/skb
bpftool cgroup attach /sys/fs/cgroup/mygrp egress pinned /sys/fs/bpf/test2 multi
My simple program with different debug print statement (test2.c):
SEC("cgroup/skb")
int cgrp_dump_pkt(struct __sk_buff *skb) {
bpf_printk("welcome test 2");
return 1;
}
ping-446804 [000] d... 374792.037025: bpf_trace_printk: welcome test 1
ping-446804 [000] d... 374792.037038: bpf_trace_printk: welcome test 2
Related
I'm trying to run Zephyr Logging by using DMA on UART on STM32L432KC.
That's simple main loop:
#include <zephyr/kernel.h>
#include <zephyr/logging/log.h>
LOG_MODULE_REGISTER(example, LOG_LEVEL_DBG);
int main(void)
{
while (1) {
LOG_INF("main loop");
k_msleep(1000);
}
}
That's full configuration:
CONFIG_SOC_SERIES_STM32L4X=y
CONFIG_SOC_STM32L432XX=y
# Enable MPU
CONFIG_ARM_MPU=y
# Enable HW stack protection
CONFIG_HW_STACK_PROTECTION=y
# enable uart driver
CONFIG_SERIAL=y
# enable GPIO
CONFIG_GPIO=y
# Enable Clocks
CONFIG_CLOCK_CONTROL=y
# console
CONFIG_CONSOLE=y
CONFIG_UART_CONSOLE=y
CONFIG_UART_ASYNC_API=y
# enable DMA
CONFIG_DMA=y
# enable pin controller
CONFIG_PINCTRL=y
# config logging
CONFIG_LOG=y
CONFIG_LOG_BACKEND_UART_ASYNC=y
CONFIG_LOG_MODE_IMMEDIATE=y
Then build it and flash:
west flash -b nucleo_l432kc
In logs I see following output:
*** Booting Zephyr OS build zephyr-v3.2.0-863-g78809549ee4c ***
[00:00:01.001,000] <inf> example: main loop
When I change CONFIG_LOG_MODE_IMMEDIATE=y to CONFIG_LOG_MODE_DEFERRED=y it prints log every second like I expect.
I discovered that code stops here on k_sem_take(&sem, K_FOREVER) and this uart_callback is never called to release this mutex.
Also discovered when following options are enabled it's working fine with CONFIG_LOG_MODE_DEFERRED=y option:
CONFIG_LOG_PROCESS_THREAD=y
CONFIG_LOG_PROCESS_THREAD_SLEEP_MS=100
But that makes me wondering why another thread must be created? DMA is no blocking. Really confusing.
What am I missing? I would like to delegate all logging to DMA buffer with Logging subsystem features, without additional threads and buffering. Is there someone experienced who can point what is wrong?
I want to create a program that logs a message when net_ns_net_exit function in the kernel is called.
SEC("fexit/net_ns_net_exit")
int BPF_PROG(net_ns_net_exit, struct net *net, long ret)
#endif
{
__u64 netns_inum = BPF_CORE_READ(net, ns.inum);
debugf("removing: netns_inum: %u", netns_inum);
return 0;
}
bpftool v7.0.0
using libbpf v1.0
features: libbfd, libbpf_strict, skeletons
I am building in on Ubuntu 22.04 with kernel 5.15.0-52-generic, AMD. The program works well on this machine, but when I try to run it on GCP with kernel version 5.10.133+ it fails with error:
libbpf: failed to find kernel BTF type ID of 'net_ns_net_exit': -3
libbpf: prog 'net_ns_net_exit': failed to prepare load attributes: -3
libbpf: prog 'net_ns_net_exit': failed to load: -3
libbpf: failed to load object 'remove_net_ns'
libbpf: failed to load BPF skeleton 'remove_net_ns': -3
loading remove_net_ns skeleton failed with error: -3
I've checked if vmlinux is available on machines and I found that both: GCP and my local ubuntu have: /sys/kernel/btf/vmlinux. I've checked the content(with vim) on the VM and I've noticed that my VM has net_ns_net_exit but GCP doesn't. I am not sure if it's related and how exactly make it works. Method: net_ns_net_exit was introduced to the kernel a long time ago so I've expected that it should be available. Maybe someone had this problem and has some advice or knows how to approach this issue?
The fexit hook points don't offer a stable API. So if the function you are trying to trace is inlined (as seems to be the case with your GCP kernel), then your only choice is to find another function that gives you the same information.
What is the correct method for using multiple CPU cores with jax.pmap?
The following example creates an environment variable for SPMD on CPU core backends, tests that JAX recognises the devices, and attempts a device lock.
import os
os.environ["XLA_FLAGS"] = '--xla_force_host_platform_device_count=2'
import jax as jx
import jax.numpy as jnp
jx.local_device_count()
# WARNING:absl:No GPU/TPU found, falling back to CPU. (Set TF_CPP_MIN_LOG_LEVEL=0 and rerun for more info.)
# 2
jx.devices("cpu")
# [CpuDevice(id=0), CpuDevice(id=1)]
def sfunc(x): while True: pass
jx.pmap(sfunc)(jnp.arange(2))
Executing from a jupyter kernel and observing htop shows that only one core is locked
I receive the same output from htop when omitting the first two lines and running:
$ env XLA_FLAGS=--xla_force_host_platform_device_count=2 python test.py
Replacing sfunc with
def sfunc(x): return 2.0*x
and calling
jx.pmap(sfunc)(jnp.arange(2))
# ShardedDeviceArray([0., 2.], dtype=float32, weak_type=True)
does return a SharedDeviecArray.
Clearly I am not correctly configuring JAX/XLA to use two cores. What am I missing and what can I do to diagnose the problem?
As far as I can tell, you are configuring the cores correctly (see e.g. Issue #2714). The problem lies in your test function:
def sfunc(x): while True: pass
This function gets stuck in an infinite loop at trace-time, not at run-time. Tracing happens in your host Python process on a single CPU (see How to think in JAX for an introduction to the idea of tracing within JAX transformations).
If you want to observe CPU usage at runtime, you'll have to use a function that finishes tracing and begins running. For that you could use any long-running function that actually produces results. Here is a simple example:
def sfunc(x):
for i in range(100):
x = (x # x)
return x
jx.pmap(sfunc)(jnp.zeros((2, 1000, 1000)))
I am writing a C program with the aim of configuring a peripheral device (the CS5368 ADC) via the I2C interface of a Dante Brooklyn II, a board based on a Microblaze soft-core processor running uClinux 2.4.
I have implemented the configuration following the Dante OEM docs for guidance, however when running my program I am encountering an "Operation not permitted" (EPERM) error when attempting to write data to I2C using i2c_smbus_write_byte_data.
Here is the section of code containing the culprit call to i2c_smbus_write_byte_data:
// Set ADC I2S to "Slave mode all speeds".
printf("Set the CS5368 I2S mode to slave\n");
unsigned char adc_dif = 0x01; // I2S mode
unsigned char adc_mode = 0x03; // Slave mode all speeds
unsigned char data = 0x90 | (adc_dif << 2) | adc_mode;
result = i2c_smbus_write_byte_data(i2c_fd, CS5368_GCTL_MDE, data);
if (result < 0) {
perror("Failed to write to the 'Global Mode Control' register");
return -1;
}
Here is the code within context of the full source of the small program. The program begins by resetting the CS5368 via a GPIO pin before doing the configuration via I2C.
EPERM is returned whether or not I have the CS5368 wired up. I've been able to successfully configure the CS5368 using the I2C interface of an Arduino Uno, so the issue does not appear to be related to the CS5368.
To run the program I login to the board via telnet as root, so I doubt the error has anything to do with user permissions.
The OEM docs state:
The Brooklyn II module can operate as an I2C controller running at
100Khz and using 7 bit addressing mode. It supports multi-master
operation. I2C devices can be accessed from user application running
on the Brooklyn II module. The interface supports the SMBus (System
Management Bus) protocol, which is a subset from the I2C protocol.
It goes on to list the supported i2c_smbus_* functions including i2c_smbus_write_byte_data, so the issue does not appear to be related to lack of support for SMBus or I2C.
I came across a related issue where a user was getting an EPERM error code when attempting to use the I2C write API, however the solution appears to have been to use the i2c_smbus_* API instead which I am already doing.
Any advice on what could be causing this error code to be returned or how to debug the issue further would be greatly appreciated!
Edit: In case it helps, here is the full output, starting from logging onto the board via telnet after having moved the exe to /tmp via ftp:
$ telnet 169.254.72.245
Trying 169.254.72.245...
Connected to 169.254.72.245.
Escape character is '^]'.
login: root
Password:
BusyBox v1.23.2 (2018-05-31 11:33:18 AEST) hush - the humble shell
/ # cd /tmp
/var/tmp # ./cs5368-i2c-config
Open GPIO device
Set GPIO tristate outputs
Set GPIO pins low
Sleep for 10 secs
Set GPIO pins high
Close GPIO file descriptor
Searching for I2C device
Opening /dev/i2c-0
Setting I2C_SLAVE 4c...
I2C Interface found: /dev/i2c-0
Set the CS5368 as the slave
Set the CS5368 I2S mode to slave
Failed to write to the 'Global Mode Control' register: Operation not permitted
I followed this article to enable FTRACE
https://lwn.net/Articles/365835/
to test a realtime system, my system uses arm cortexa15 (Description: https://mp.renesas.com/en-us/rzg/marketplace/board/RZGB000003.html)
CONFIG_FTRACE=y
CONFIG_FUNCTION_TRACER=y
CONFIG_FUNCTION_GRAPH_TRACER=y
CONFIG_STACK_TRACER=y
CONFIG_DYNAMIC_FTRACE=y
But, it didn't work, caused the system hang-up when starting kernel.
Even referred How to Enable or configure ftrace module
I would like to test latency in the realtime system with cyclictest (option -b to trigger FTRACER)
cyclictest -a -t -n -p99 -f -b100
It generated dump message:
INFO: debugfs mountpoint: /sys/kernel/debug/tracing/
WARN: tracing_enabled or tracing_on not found
debug fs not mounted, TRACERs not configured?
could not set ftrace_enabled to 0
FATAL: Can't open /sys/kernel/debug/tracing/available_tracers for reading
I repeated a next step to enable a group of tracer configs:
CONFIG_FTRACE=y
CONFIG_DYNAMIC_FTRACE=y
CONFIG_HAVE_DYNAMIC_FTRACE=y
CONFIG_HAVE_FUNCTION_TRACER=y
CONFIG_HAVE_FUNCTION_GRAPH_TRACER=y
CONFIG_HAVE_FTRACE_MCOUNT_RECORD=y
CONFIG_HAVE_SYSCALL_TRACEPOINTS=y
CONFIG_HAVE_C_RECORDMCOUNT=y
CONFIG_TRACING_SUPPORT=y
CONFIG_FUNCTION_TRACER=y
CONFIG_IRQSOFF_TRACER=y
CONFIG_FUNCTION_PROFILER=y
CONFIG_PREEMPT_TRACER=y
CONFIG_SCHED_TRACER=y
CONFIG_ENABLE_DEFAULT_TRACERS=y
CONFIG_FTRACE_SYSCALLS=y
CONFIG_TRACER_SNAPSHOT=y
CONFIG_STACK_TRACER=y
CONFIG_BLK_DEV_IO_TRACE=y
CONFIG_TRACEPOINT_BENCHMARK=y
CONFIG_BACKTRACE_SELF_TEST=y
CONFIG_EARLY_PRINTK=y
CONFIG_DEBUG_LL=y
The result still was the same. Kernel hung and didn't show anything.
Anyone who deal with realtime system and Ftrace can help ? Thanks.
I resolve my problem. Below is a part of my defconfig file.
CONFIG_STACKTRACE_SUPPORT=y
CONFIG_TRACE_IRQFLAGS_SUPPORT=y
CONFIG_TRACEPOINTS=y
CONFIG_STACKTRACE=y
CONFIG_NOP_TRACER=y
CONFIG_TRACER_MAX_TRACE=y
CONFIG_TRACE_CLOCK=y
CONFIG_CONTEXT_SWITCH_TRACER=y
CONFIG_GENERIC_TRACER=y
CONFIG_FTRACE=y
CONFIG_FUNCTION_TRACER=y
CONFIG_FUNCTION_GRAPH_TRACER=y
CONFIG_SCHED_TRACER=y
CONFIG_FTRACE_SYSCALLS=y
CONFIG_TRACER_SNAPSHOT=y
CONFIG_STACK_TRACER=y
CONFIG_DYNAMIC_FTRACE=y
CONFIG_FTRACE_MCOUNT_RECORD=y
After enabling Ftrace tool, the culprit can then be found in the trace output at /sys/kernel/debug/tracing/trace. The kernel function that was executed just before a latency of more than 100 microseconds was detected is marked with an exclamation mark.