Here is TAS code
boolean TAS(*target)
{
boolean rv = *target; //Save the initial value
*target = true; //Set lock
return rv; //Return initial value
}
The following codes means two different tries for the increment operation on a thread safe counter using tas to maintain the counter value consistency.
code 1.
Counter::Increment() {
while (TAS(&lock)) ;
value++;
lock = FREE;
memory_barrier();
}
code 2.
Counter::Increment() {
while (lock == BUSY && TAS(&lock)) ;
value++;
lock = FREE;
memory_barrier();
}
What will be the issue of each codes that cannot handle consistency
Assuming that FREE == false and BUSY == true, here is a flow which breaks mutual exclusion:
Start thread 1:
Increment
TAS
rv = *lock; // FREE
Switch to thread 2:
Increment
TAS
rv = *lock; // FREE
*lock = BUSY;
return rv; // FREE
Load value 0 from memory and push it into the stack
Switch to thread 1:
*lock = BUSY;
return rv; // FREE
Load value 0 from memory and push it into the stack
Push 1 into the stack
Pop two integers from the stack and add them
Store the result (0 + 1) into value
Switch to thread 2:
Push 1 into the stack
Pop two integers from the stack and add them
Store the result (0 + 1) into value
Related
This is rather ugly. We're getting weird hangups caused (read: triggered) by some security scanning software. I suspect it has a nonstandard TCP/IP stack, and the customer is asking why hangups.
Static analysis suggests only two possible locations for the hangup. The hangup either has to be in ReadFile() or WriteFile() on a socket; and WriteFile() cannot hang here unless the scanner is designed to make WriteFile() hang by setting the window size to zero. If WriteFile() were to return at all even if it didn't make progress I'd be able to knock the thing out of its wedged state. I also don't think the log state is consistent with WriteFile() returning.
So onto ReadFile(): this is the calling sequence:
SOCKET conn;
HANDLE unwedgeEvent = CreateEvent(NULL, TRUE, FALSE, NULL);
HANDLE listenEvent = CreateEvent(NULL, FALSE, FALSE, NULL);
if (listenEvent == NULL) return;
//...
conn = accept(lstn);
for (;;) {
HANDLE wakeup[2];
wakeup[0] = unwedgeEvent;
wakeup[1] = listenEvent;
if (WSAEventSelect(conn, socket, FD_READ | FD_CLOSE) == 0) {
// Log error
break;
}
which = WaitForMultipleObjects(2, wakeup, FALSE, INFINITE);
if (which < 0) {
// Log error
break;
}
if (which == 1) {
DWORD read;
r = ReadFile(conn, chunk, 4096, &read, NULL);
if (r == 0) {
// Handle error -- not stuck here
} else {
// Handle data -- not stuck here either
}
}
if (which == 0) break;
}
Where signalling unwedgeEvent doesn't manage to accomplish anything and the thread remains stuck forever.
So the real question is have I gone nuts or is this really a thing that can happen?
So this has gone somewhat off the deep end; I don't need non-blocking sockets at all. I need a select() that takes handle arguments to things that are sockets and things that are not sockets.
The following API sequences do not hang in ReadFile:
---------------------------------------------------------------
Sender B Receiver
WSAEventSelect
* WaitForMultipeObjects
send(buffer size = 1)
ReadFile(size = 1)
WSAEventSelect
* WaitForMultipeObjects
send(buffer size = 1)
ReadFile(size = 1)
................................................................
WSAEventSelect
* WaitForMultipeObjects
send(buffer size = 2)
ReadFile(size = 1)
WaitForMultipeObjects
ReadFile(size = 1)
................................................................
WSAEventSelect
* WaitForMultipeObjects
send(buffer size = 1)
send(buffer size = 1)
ReadFile(size = 1)
WaitForMultipeObjects
ReadFile(size = 1)
................................................................
WSAEventSelect
send(buffer size = 1)
WaitForMultipeObjects
send(buffer size = 1)
ReadFile(size = 1)
WaitForMultipeObjects
ReadFile(size = 1)
................................................................
WSAEventSelect
send(buffer size = 1)
WaitForMultipeObjects
send(buffer size = 1)
ReadFile(size = 2)
* WaitForMultipeObjects
send(buffer size = 1)
ReadFile(size = 1)
................................................................
WSAEventSelect
send(buffer size = 1)
WaitForMultipeObjects
ReadFile(size = 1)
send(buffer size = 1)
WaitForMultipeObjects
ReadFile(size = 1)
I see a number of issues with your code:
You should use WSACreateEvent() and WSAWaitForMultipleEvents() instead of CreateEvent() and WaitForMultipleObjects(). Although the current implementation is that the former APIs simply map to the latter APIs, Microsoft is free to change that implementation at any time without breaking code that uses the former APIs properly.
In your call to WSAEventSelect(), socket should be listenEvent instead.
WSAEventSelect() returns SOCKET_ERROR (-1) on failure, not 0 like you have coded.
You are not calling WSAEnumNetworkEvents() at all, which you need to do in order to determine if FD_READ was the actual type of event triggered, and to clear the socket's event state and reset the event object. So, you may be acting on a stale read state, which could explain why you end up calling ReadFile() when there is actually nothing available to read.
WSAEventSelect() puts the socket into non-blocking mode (per its documentation), so it is actually not possible for ReadFile() (or any other reading function) to block on the socket. However, it could fail immediately with a WSAEWOULDBLOCK error, so make sure you are not treating that condition as a fatal error.
The WSAEventSelect() documentation does not list ReadFile() as a supported function for re-enabling events when calling WSAEnumNetworkEvents(). Although the Socket Handles documentation does say that a Winsock SOCKET can be used with non-Winsock I/O functions like ReadFile(), it recommends that a SOCKET should only be used with Winsock functions. So, you should use send()/recv() or WSASend()/WSARecv() instead of WriteFile()/ReadFile().
With that said, try something more like the following:
HANDLE unwedgeEvent = CreateEvent(NULL, TRUE, FALSE, NULL);
if (unwedgeEvent == NULL) {
// Log error
return;
}
...
SOCKET conn = accept(lstn, NULL, NULL);
if (conn == INVALID_SOCKET) {
// Log error
return;
}
...
WSAEVENT listenEvent = WSACreateEvent();
if (listenEvent == NULL) {
// Log error
}
else if (WSAEventSelect(conn, listenEvent, FD_READ | FD_CLOSE) == SOCKET_ERROR) {
// Log error
}
else {
WSAEVENT wakeup[2];
wakeup[0] = (WSAEVENT) unwedgeEvent;
wakeup[1] = listenEvent;
char chunk[4096];
int read;
do {
DWORD which = WSAWaitForMultipleEvents(2, wakeup, FALSE, WSA_INFINITE, FALSE);
if (which == WSA_WAIT_FAILED) {
// Log error
break;
}
if (which == WSA_WAIT_EVENT_0) {
break;
}
if (which == (WSA_WAIT_EVENT_0+1)) {
WSANETWORKEVENTS events = {};
if (WSAEnumNetworkEvents(conn, listenEvent, &events) == SOCKET_ERROR) {
// Log error
break;
}
if (events.lNetworkEvents & FD_READ) {
read = recv(conn, chunk, sizeof(chunk), 0);
if (read == SOCKET_ERROR) {
if (WSAGetLastError() != WSAEWOULDBLOCK) {
// Log error
break;
}
}
else if (read == 0) {
break;
}
else {
// Handle chunk up to read number of bytes
}
}
if (events.lNetworkEvents & FD_CLOSE) {
break;
}
}
}
while (true);
WSACloseEvent(listenEvent);
}
closesocket(conn);
I have frames that are delimited by bytes to start and stop the frame (they do not appear in the stream).
I read a chunk from disk or network socket, i then need to pass to a deserializer but only after I have de-framed the packet first.
Frames may span multiple chunks that have been read, note how frame 3 is split across array 1 and array 2.
Rather than reinvent the wheel for this common problem, do any github or similar projects exist?
I am investigating ReadOnlySequenceSegment<T> from https://www.codemag.com/article/1807051/Introducing-.NET-Core-2.1-Flagship-Types-Span-T-and-Memory-T and will post updates as I work out the requirements.
Update
Further to Stephen Cleary link (thank you!!) to https://github.com/davidfowl/TcpEcho/blob/master/src/Server/Program.cs I have the below.
My data is json, so unlike the original question the delimiter tokens will appear in the stream. Therefore I have to count the array delimitator and only declare a frame when i have found the outermost [ and ] characters.
The below code works, and less manual copies done (not sure if still done behind the scenes - code is quite neater using David Fowl approach).
However I am casting to array instead of using buffer.PositionOf((byte)'[') since I was unable to see how I could call the PositionOf with an offset applied (i.e. scan deeper into the frame past previously found delimiter tokens).
Am i using/butchering the library in a brute force way, or is the below good to go with the array cast?
class Program
{
static async Task Main(string[] args)
{
using var stream = File.Open(args[0], FileMode.Open);
var reader = PipeReader.Create(stream);
while (true)
{
ReadResult result = await reader.ReadAsync();
ReadOnlySequence<byte> buffer = result.Buffer;
while (TryDeframe(ref buffer, out ReadOnlySequence<byte> line))
{
// Process the line.
var str = System.Text.Encoding.UTF8.GetString(line.ToArray());
Console.WriteLine(str);
}
// Tell the PipeReader how much of the buffer has been consumed.
reader.AdvanceTo(buffer.Start, buffer.End);
// Stop reading if there's no more data coming.
if (result.IsCompleted)
{
break;
}
}
// Mark the PipeReader as complete.
await reader.CompleteAsync();
}
private static bool TryDeframe(ref ReadOnlySequence<byte> buffer, out ReadOnlySequence<byte> frame)
{
int frameCount = 0;
int start = -1;
int end = -1;
var bytes = buffer.ToArray();
for (var i = 0; i < bytes.Length; i++)
{
var b = bytes[i];
if (b == (byte)'[')
{
if (start == -1)
start = i;
frameCount++;
}
else if (b == (byte)']')
{
frameCount--;
if (frameCount == 0)
{
end = i;
break;
}
}
}
if (start == -1 || end == -1) // no frame found
{
frame = default;
return false;
}
frame = buffer.Slice(start, end+1);
buffer = buffer.Slice(frame.Length);
return true;
}
}
do any github or similar projects exist?
David Fowler has an echo server that uses Pipelines to implement delimited frames.
The question is specific to a pattern that Flawfinder reports:
The snippet
unsigned char child_report;
...
auto readlen = read(pipefd[0], (void *) &child_report, sizeof(child_report));
if(readlen == -1 || readlen != sizeof(child_report)) {
_ret.failure = execute_result::PREIO ; // set some flags to report to the caller
close(pipefd[0]);
return _ret;
}
...
int sec_read = read(pipefd[0], (void *) &child_report, sizeof(child_report));
child_report = 0; // we are not using the read data at all
// we just want to know if the read is successful or not
if (sec_read != 0 && sec_read != -1) { // if success
_ret.failure = execute_result::EXEC; // it means that the child is not able to exec
close(pipefd[0]); // as we set the close-on-exec flag
return _ret; // and we do write after exec in the child
}
I turned out that Codacy (therefore flawfinder) reports such issues on both read:
Check buffer boundaries if used in a loop including recursive loops (CWE-120, CWE-20).
I don't understand.
There is no loop.
In the second case we are not using the read data at all
This is not typical C string, and we don't rely on the ending '\0'
Is there any flaw that I'm not aware of in the code?
I finally conclude this should be a false positive. I check Flawfinder's code and it seems that it is basically doing pattern matching.
https://github.com/david-a-wheeler/flawfinder/blob/293ca17d8212905c7788aca1df7837d4716bd456/flawfinder#L1057
I'm learning about osdev and looking up xv6 code, currently - the console code in particular. Basically, I don't get how the console launches a process.
in console.c there is a function:
void consoleintr(int (*getc)(void)) {
....
while((c = getc()) >= 0) {
switch(c) {
....
default:
....
if(c == '\n' || c == C('D') || input.rightmost == input.r + INPUT_BUF) {
wakeup(&input.r);
}
}
}
So I get it, when the line ends (or the length of the buffer exceeds maximum), it launches wakeup(&input.r)
Then there is this in proc.c:
// Wake up all processes sleeping on chan.
// The ptable lock must be held.
static void wakeup1(void *chan)
{
struct proc *p;
for(p = ptable.proc; p < &ptable.proc[NPROC]; p++)
if(p->state == SLEEPING && p->chan == chan)
p->state = RUNNABLE;
}
// Wake up all processes sleeping on chan.
void wakeup(void *chan)
{
acquire(&ptable.lock);
wakeup1(chan);
release(&ptable.lock);
}
What is happening here? Why is it comparing address of a console buffer and proc's chan? What is this chan?
It is for processes who waiting (sleeps) for console input. See here:
235 int
236 consoleread(struct inode *ip, char *dst, int n)
...
251 sleep(&input.r, &cons.lock);
The code you have mentioned wakeups this sleeping processes, because data have came from console and is available now for processing.
chan - is a channel. You can wait (sleep) for different things. This channel is for console input data.
In the shared buffer memory problem , why is it that we can have at most (n-1) items in the buffer at the same time.
Where 'n' is the buffer's size .
Thanks!
In an OS development class in college, I had an adjunct teacher that claimed it was impossible to have a software-only solution that could use all N elements in the buffer.
I proved him wrong with something I decided to call the race track solution (inspired by the fact that I like to run track).
On a race track, you are not limited to a 400 meter race; a race can consist of more than one lap. What happens if two runners are neck and neck
in a race? How do you know whether they are tied, or whether one runner has lapped the other? The answer is simple: in a race, we don't monitor a runner's position
on the track; we monitor the distance each runner has traversed. Thus, when two runners are neck and neck, we can disambiguafy between a tie and when one runner has
lapped the other.
So, our algorithm has an N-element array, and manages a 2N race. We don't restart the producer/consumer's counter back to zero until they finish their respective 2N race.
We don't allow the producer to be more than one lap ahead of the consumer, and we don't allow the consumer to be ahead of the producer.
Actually, we only have to monitor the distance between the producer and consumer.
The code is as follows:
Item track[LAP];
int consIdx = 0;
int prodIdx = 0;
void consumer()
{ while(true)
{ int diff = abs(prodIdx - consIdx);
if(0 < diff) //If the consumer isn't tied
{ track[consIdx%LAP] = null;
consIdx = (consIdx + 1) % (2*LAP);
}
}
}
void producer()
{ while(true)
{ int diff = (prodIdx - consIdx);
if(diff < LAP) //If prod hasn't lapped cons
{ track[prodIdx%LAP] = Item(); //Advance on the 1-lap track.
prodIdx = (prodIdx + 1) % (2*LAP);//Advance in the 2-lap race.
}
}
}
It's been a while since I originally solved the problem, so this is according to my best recollection. Hopefully I didn't overlook any bugs.
Hope this helps!
Oops, here's a bug fix:
Item track[LAP];
int consIdx = 0;
int prodIdx = 0;
void consumer()
{ while(true)
{ int diff = prodIdx - consIdx; //When prodIdx wraps to 0 before consIdx,
diff = 0<=diff? diff: diff + (2*LAP); //think in 3 Laps until consIdx wraps to 0.
if(0 < diff) //If the consumer isn't tied
{ track[consIdx%LAP] = null;
consIdx = (consIdx + 1) % (2*LAP);
}
}
}
void producer()
{ while(true)
{ int diff = prodIdx - consIdx;
diff = 0<=diff? diff: diff + (2*LAP);
if(diff < LAP) //If prod hasn't lapped cons
{ track[prodIdx%LAP] = Item(); //Advance on the 1-lap track.
prodIdx = (prodIdx + 1) % (2*LAP);//Advance in the 2-lap race.
}
}
}
Well, theoretically a bounded buffer can hold elements upto its size. But what you are saying could be related to certain implementation quirks like a clean way of figuring out when the buffer is empty/full. This question -> Empty element in array-based bounded buffer deals with a similar thing. See if it helps.
However you can of course have implementations that have all n slots filled up. That's how the bounded buffer problem is defined anyway.