Copy a file from client to server via rest end point using golang - rest

I want to copy a file from the client to the server through the rest end point exposed by the server. I referred the various questions and answers in stackoverflow but I could not get a clear picture of it.
I just want a sample client and server code in golang to copy the file from client and save it on the server.
Thanks in advance.

Direction: Server to Client
So -- both sides are in Go? Okay, let's start with the server side. See my WebLoad.go file from my CSVStorageServer server: (Link to Github)
At line 17, I define the handler for the web server. This method will build a zip file on-demand and send it to the browser. The important part regarding to your question are line 77 up to 82. Here, I set the headers for the client, e.g. content length and type. Line 82 sends the whole data to the client side. It copies the bytes from the on-demand zip file to the wire.
On the client side, you trigger e.g. a GET request and store the result. Here an example: https://golang.org/pkg/net/http/#example_Get
With http.Get(... you trigger the GET request. With ioutil.ReadAll(res.Body) you read all bytes from the server and store it to a variable. Afterwards, you could write the bytes to the disk or process it in-memory.
I hope, this answer helps you.
Best regards, Thorsten
Edit #1:
Regarding the REST end-point, cf. the server definition (link to Github). Line 16 defines the REST end-point for this handler. In this case, it gets available as /load. You could use any REST-like path here, e.g. /open/file/USERID/send, etc.
Direction: Client to Server
In order to copy a file from client to server side, similar operations are necessary. On the client side, a POST request is necessary as multipart/form-data. Here is a good example for this: Link to a blog post. This example considers also the server part. The relevant client part is the function func postFile(filename string, targetUrl string) error { ... }.
For the server part, here an own example: Link to Github. This example receives an file from the client and writes it to a MongoDB database. The relevant parts are:
Line 39 read the file from the client: file, fileHeader, fileError := request.FormFile("file") The result is a handle to this uploaded file.
Line 60 copies all bytes from the source (browser or Go client) into a destination (here, the MongoDB): _, errCopy := io.Copy(newFile, file).
Edit #2:
Here is a full working example: https://github.com/SommerEngineering/Example010 where client and server are in the same program. It should be easy to split it into two programs.

Related

How do I translate the following POST request into ESP8266 AT-command format?

I've got a working local website that takes in HTML form data.
The fields are:
Temperature
Humidity
The server successfully receives the data and spits out a graph updated with the new entries.
Using a browser tool, I was able to capture the actual POST request as follows:
http://127.0.0.1:5000/add_data
Temperature=25.4&Humidity=52.2
Content-Length:30
Now, I want to migrate from using the human interface browser with manual entries to an ESP01 device using AT commands.
According to the ESP AT-commands documentation, a POST request is performed using the following command:
AT+HTTPCPOST=
Find the link below for the full description of the command.
I cannot seem to get this POST request working. The ESP01 device immediately returns an "ERROR" message without any delay, as though it did not even try to send the request, that the syntax might be wrong.
Among many variations, the following is my best attempt:
AT+HTTPCPOST="http://MYIPADDR:5000/add_data",30,2,"Temperature: 25.4","Humidity: 52.2"
With MYIPADDR above replaced with my IP address.
How do I translate a post request into ESP01 AT command format, and are there any prerequisites needed to be in place to perform such a request?
I did connect the ESP01 device to the WiFi network.
Here's the link to the POST AT command description:
https://docs.espressif.com/projects/esp-at/en/release-v2.2.0.0_esp8266/AT_Command_Set/HTTP_AT_Commands.html#cmd-httpcpost
The documentation says:
AT+HTTPCPOST=url,length[,<http_req_header_cnt>][,<http_req_header>..<http_req_header>]
Response:
OK
The symbol > indicates that AT is ready for receiving serial data, and you can enter the data now. When the requirement of message length
determined by the parameter is met, the transmission starts.
...
Parameters
: HTTP URL. : HTTP data length to POST. The maximum
length is equal to the system allocable heap size.
<http_req_header_cnt>: the number of <http_req_header> parameters.
[<http_req_header>]: you can send more than one request header to the
server.
You're sending:
AT+HTTPCPOST="http://MYIPADDR:5000/add_data",30,2,"Temperature: 25.4","Humidity: 52.2"
The length is 30. The problem is that everything after the length is HTTP header fields; you need to send the variables in the body. So the command is:
AT+HTTPCPOST="http://MYIPADDR:5000/add_data",30
followed on the next line by after the ESP-01 send the > character:
Temperature=25.4&Humidity=52.2
Because you passed 30 as the body length, the ESP-01 will read exactly 30 characters after the end of the AT command and send that data as the post body. If the size of that data changes (for instance, maybe the temperature is 2.2, so one digit less), you'll need to send the new length rather than 30.

Bidirectional communication of Unix sockets

I'm trying to create a server that sets up a Unix socket and listens for clients which send/receive data. I've made a small repository to recreate the problem.
The server runs and it can receive data from the clients that connect, but I can't get the server response to be read from the client without an error on the server.
I have commented out the offending code on the client and server. Uncomment both to recreate the problem.
When the code to respond to the client is uncommented, I get this error on the server:
thread '' panicked at 'called Result::unwrap() on an Err value: Os { code: 11, kind: WouldBlock, message: "Resource temporarily unavailable" }', src/main.rs:77:42
MRE Link
Your code calls set_read_timeout to set the timeout on the socket. Its documentation states that on Unix it results in a WouldBlock error in case of timeout, which is precisely what happens to you.
As to why your client times out, the likely reason is that the server calls stream.read_to_string(&mut response), which reads the stream until end-of-file. On the other hand, your client calls write_all() followed by flush(), and (after uncommenting the offending code) attempts to read the response. But the attempt to read the response means that the stream is not closed, so the server will wait for EOF, and you have a deadlock on your hands. Note that none of this is specific to Rust; you would have the exact same issue in C++ or Python.
To fix the issue, you need to use a protocol in your communication. A very simple protocol could consist of first sending the message size (in a fixed format, perhaps 4 bytes in length) and only then the actual message. The code that reads from the stream would do the same: first read the message size and then the message itself. Even better than inventing your own protocol would be to use an existing one, e.g. to exchange messages using serde.

How is the best way GWT client load data from server file

my GWT client needs to load data from a file located in the server exactly under war directory what is the best solution???.
The data are double values.
I know the client at the end is converted to java script.
As per my understanding, you want to read the data from file!! is it so?
if it is true, you have to create servlet to read data from file. Then make a request to servlet from client. Let say example
1) ReadFileServlet.java: This is server side code that contains code to read file from location.
2) ReadFileGWT.java: This is gwt file. From here, you make request to call ReadFileServlet file.
Hope it helps!!

How do I decode a websocket packet?

I'm using Wireshark packet analyzer & when I filter for all "Websocket" packets I see what I am sending /receiving to the host. When I check individual packets mine always show as [MASKED], but you can 'Umask Payload' which shows the data in clear text that looks like this:
<IC sid="52ccc752-6080-4668-8f55-662020d83979" msqid="120l93l9l114l30l104"/>
However, if I 'Follow TCP stream & look at that same packet, the data shows up as encoded in some way like this:
....K#....../...y#..|...}...f...s...~...}...{G..r...kN.."G..z...r...'...'...z...d.
The problem is all Websocket packets I receive from the host come as encoded, it is NOT SSL & I can't figure out how to decode them, I have no idea what they are even encoded as (but yet my browser can decode it).
I assume that whatever method they are coming back to me as encoded data is the same method that my data is encoded when I use 'Follow TCP stream'.
Can someone please help me figure out how to decode the data the host is sending me? See host data below
~.^jVpZc9y4Ef4ryFQ5+yJpeB+JJJdmNPI6G++mrN249kkFkuChIQmG5Fgj//p0AyAJypzxyi5T6P76
QKPRuHz9cUu6IrlZuVYcx75rXXpGYFw6nhdcBqnrXnqeZVhGEtihH65Id7NKWEoPZb8iVfc/FDQt
owztMixN0yltozQNZ3V7ncZkbxrAXZE8vFkZK2g66msJchLIjyuoiWQmvvyApGUY+JsJKPGLkrIF
IHcFALVJNXtTWsl9adMDPlAtQ1AZME0XvoFsShDz5McVn0J6y2z5ceTHlB9pnEltheQVEllIXiGR
z7Ifz6Cz4c2h6XkDLTDUFlkOQYuk/5EUimTnIykUyc5HoeTJjlHVMgWPwifv++Yf6/XLy8uVadlp
Sbs8zml/FfMKAKA4Z2WzLuqEHa/yviqBCEZXJJXeUzC25c2rcIhAEM1LyzBt8jtvtp8+kUee9i+0
ZWTL2+aKkLuyJJ8R2pHPrGPtV5ZcgRIXNVLoF6vh62tpkToy9LIzexnxvRydWEY8lhGPZRxjGccY
IDBEezkMsZSLlRZLtmQQYhm8WCBvr2lAMhFVyDqPpKDmPy1Pi5KtSGaM4Xrlh/aFRV3Rs3Uj+VdN
3rw/QJ9u3v3xuPv8DhSsUw/+ocHtdeKRDNz0wF4GfjpesJrM+CQDx5ACHtFkHdG6Zq159dxkQLPO
jxFa8Ucl7hsl1l9Sss5518vRPa/Ovupe0r+i7qXnTzT5ytq+6Io6e5LiSybMtzacUzbK4ivDZFzo
tmm8UeL+NUeBAKNYsa5jdcbay5TR/tCymZ/rBAYxCbWsuP2ZlSUn7/787Y/Pv9592r27IB9/qsi7
T3+KFklbXpHu0DS87UnPaHVBICKkoq/kI8EeEEif9zI7UFsxU/UCzpGEI4bUjCUT8AsrwWlGek6e
eVGTQ3dBNFHyN5VwSQhwc3I4kA4DN5Ct4oL0OWvZ3yYS+IfTFI0moDt7P2Pl9KvkRYzVGgvI9U89
6YAq2ClvCc1YNyn+gnYm+bxIEsD2kHCMJPS1e080KO1/6sih+Z6W06ZhNbr1HatmL5ND9+g6yThP
wASt950KJ434oUcH2o6V6YT/lcMAcU4imlwQWifDwEjuXUW/gb6pMx9ayI+piYOeSIvIuBoZW34o
EwxMxOBv37N2EvrXoYOAcfg74T+Squg6ESDgVIc413kll8GbaB+E29DPkfI7LfdIkip4PWEfmYx8
ScENzUXax/nEmPzbvDKFWqcmUCxRuBxjqFy+O1WudWoBwiY5TD0Hlkmojz585KKkVVExRaGKYzV9
rGQtBRExF1nF+4LXa5iv6w55auZ6b/h9fqgiDXE7TAuh3ZfK/8uroj+h/CvyziqfEIvKH5sifiUP
kFyn3EfAefdHxNxCqCyUjDWvJ7R/+/btrO6BP9fsKM05j/en3EbeebdHxFy5pbR/weyj5J5nJ0y8
AOCshREwN4B1XSzBOe/5Cd0N8s4qnxBvtKuw/0yr6nR4csk9a0HHLMYfKgdLTo1sJphnDWiQuX5X
6n+gRXtKfYq8s9onxFx5MMSnaV7Jpmj7HEr2CSuRYp81NAMt2trmjLWwBpywEiP70Jw1omPeDPhg
5GSs4h9EKl6M05Cmd3V2UjNF3lndE2JxiH+pYf/TndC+F8yz6jXIYvH5Nz1k+Qn1JfLOap8Qc+We
ch7Wt1OuA+u84wNgOeiPL7ACnyptyDtf2kbEYjp+grX0hO4Kl9lzqkfAYkXYwJZhT/44legRsp9+
kOkz0NyKvRrOLg3vaHmqdir2+fKpgxZXRhxd2OjkcEok20Pb0+JU0GLJ/eGYv8UtDg6sCjWDulSe
7B4CniIA/Gh90GHLtvietafMIO+8hRHxJofVpuj3HPt6Mo17ZJ81MCGWty6HumOnNkadYJ6fJRNk
sXY8yOuI5eUHeeeXnxGxHJ0t5/uCkQ9Fe2qgY4E4n1ETZDiYpzaYckkWSlMZXpEYF6Z5YVoXpn1h
Oheme2F6citrCjH39jp39B2uOd0TfKBdI05AePRri07f5Xb8UCfrDBBXVWNr/RSCRZaV36NzOO0u
oB/hGJnjeW5BAKgLEls8NM7qmMIfWlhwvsf/UsR7OEVULOLJK4GT03eSe0AsCkIdGAQXhGAyL/Qn
hipWfYfuBHkB/yd9qWWY3+6WpeAr8JfM1LydzzAJx22zhl7nzu114ZK9J4cYciI6RBEOT+GpLCgg
C55N8jy7XLjES4VLPBAfmLw8G09Jz3COKnyyN2XaFKAzO7Cux3tGeQdVyAutZ3mn9jwIFv7t9d4y
yd4CU5sVNmxowHnstzSd3UcV/aGGxR02aqWwvj50a2is9RuPdZG4pm/aoed478vuxvw7rZp/Vv1N
gLZANep3FvR3YKApYdcGB+tM6e963jI00a0TBqW67N4XyQ3sI6/Q1Cce4TltIxU74l8TIxzfXncx
yfE67hg+bOytq250jw/iR97FGsdduLt/gNKbOwIpfuR1rHF0LN+59+WtrCaHkTxuLfwjj6LG0RY/
6kR6NIwwxGNsbgkLAVjwVehHZCHkNg/37m4rRwrlPA/DUfgzpKd7VrjSl/BO8BzZCsSlc2HP5KwZ
T3gd24YbYKn2dGTq6/1Lg1lrFjMqWnd3Gx/jSc0ZT9i7e9ia2x3ICd7O2W0eDHmTaxxT8aNudI+g
I8ToUmfGs/URo6K327t7f4clWvYofNg+7OQtrtYHV9dC/ZmcM/Mz0PuQzUYlE3LW1ts6d3JOajxh
3XZtx7bk5DaOgRVu7h5kAmstT/clc/UeZTLPdtBy1MwdPUuEL574kQ8NqMXzxDvCrH+JylbbNobX
B8gewxBaVLYa8iHCmPHMWcsask4g7YGHecaEL5bnBRtPPUCMkWCz/GRqHDwPdTJX7xETOr3dxtxt
5OOEcfTFj3yjgDEy7u8fQvlUoekUXj84MNb36nHiaO683b16o5gi8SzkAtsW9p5lPL3At/BRQ2Wy
729hjj2Hw9wUcrMxehY9cu3A2NjqseMY2hv/fiPfPMBP7z7EmRrN+ideQEavn2e1QDyLTBbMGW+W
u5HMQce7uzPkA4rWmus09QzZ23qP9o4ut1dRklmwn/V27+tREk8dR8f0/PBePnVM2RMLnYbpbjAL
4ll9iUM9l/bCszTxUsNTzyCCJ/y09JjFvs6LVQ3B/JFPJJoF0XfL9bydhwW+w6IuZklcMto+wYZB
7sRwqiiSOinhdIGFnZelhnIm2gDzJ1KaalBvTh/g4BBvcK+n2uA8bMC+0h529WAdVv5qOOPg5NJ5
UTlurXGm5QUubOqcjktRy3kFW98eHzFGP3BVmjGUYVyDMlqxp4ZmQr0iG4ocg9UM1D8V8ShiK95I
sAbCYA3nfA0brhqp0V5jBG8YSgUWArG5mQUPC8JEHaDuQKw15BQIjehoyEEa+suORa+hLE10QBkD
ShFwYau6rKF46lLHc1zeovIgB6VkqdKH69xIbqeTBNaCuOTdFDesB+KNVDWxY6xOol7rGNYFJTWS
wBfxOqCaniY2aA7HlFEE8LXkNBEPWyp5cAnGlCih85OkoRNHGxA13CQVdXPoR3kTdWbqIhMXbdHb
ST90WHcKF3I5FWZhxFV71mVc1IcAKghEJuGTk7hgi/Yggqs0vmNiWikKdLihXffC20QdsnBc9lmL
CTE87kmGr4ZhM9y7gGzLMogAawcaVrOqqIvRh1jNtKfuUCmKmmJP/GXUjYVO0MrxPharXUWPk/NY
4gRh0OwPpkbEvDoooqVU67UCC92I3A/9sUTyPqk6FE2+4N5D44x3N8Y0lk8R73teTQdV2EpHfHMA
4nCX5H6vXisXuE/R2XQ0bs8YCYRNnXVTW+wrh9EaqI7Ym92P7+wAEWcgZ+K18lJCHLzJaqbFMMeT
9CTwyOgcZdpLqEaew3SgFSwAN5wqxyL4bQHwiVHop4RU4vclc3A2Ge11srEIg0nPaJwPQNVc8usV
X0/J2NuO0a5ImI4UVwuPv3z89enu8+ffvpCvFKYgFJq2nXyc2LiG3l7H+R6yH/9PjGGw0LfTyLGN
0Ehdzw4DM6apZaXM8pMA/+NMfLNyXc+IAwcgIaNBEDhmStPYMSPT9izHSMA5QCUA8tMoCFxGHWpa
phn5lhVYNAlTljoueiItwy8ft7f/Bw==
Client to server data is XORed with a mask (included in the dataframe). Some people suggest this is in order to throw off bad caching mechanisms responding to new websocket requests with server messages from older sessions. The masking makes sure that even messages containing identical data will appear differently to applications that do not understand websockets.
Also note that there are many different size options for the headers themselves.
Refer to RFC 6455 Section 5 which defines the masking/unmasking process for payloads sent from the client to the server.
https://www.rfc-editor.org/rfc/rfc6455
If you find any freeware VBA code to do the job of forming packets let me know! :-)

syslog-ng with unix-stream destination

I am trying to configure syslog-ng destination path to use unix-stream sockets for Inter process communication. I have gone throgh this documentation http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.3-guides/en/syslog-ng-ose-v3.3-guide-admin-en/html/configuring_destinations_unixstream.html .
My syslog.conf(only part of it) for the same is as follows:
source s_dxtcp { tcp(ip(0.0.0.0) port(514)); };
filter f_request {program("dxall");};
destination d_dxall_unixstream {unix-stream("/var/run/logs/all.log");};
log {source(s_dxtcp); filter(f_request); destination(d_dxall_unixstream);};
When I restart my syslog-ng server, I have got the following message:
Connection failed; fd='11', server='AF_UNIX(/var/run/logs/all.log)',
local='AF_UNIX(anonymous)', error='Connection refused (111)'
Initiating connection failed, reconnecting; time_reopen='60'
What this error signifies? How can I use unix sockets with syslog-ng? Could any one help me out.
Till now I am not able to create a Unix Domain Socket for inter process communication. But I got a way around it. All I want is a one way communication to send data created at syslog-ng to a running java program(a process, I can say). This I achieved with Using Named Pipes in Syslog-ng. Documents for achieving is http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.4-guides/en/syslog-ng-ose-v3.4-guide-admin/html-single/index.html#configuring-destinations-pipe .
Reading from Named Pipe is same as reading from a normal file. One important point to note is that Reader process(here the Java program) should be started before Syslog-ng, (Writer, that writes log messages to the Named pipe).
Reason, Writer will block until there is a Reader. Absence of Reader will lead to loss of some messages, that got accumulated before Reader Started. And there should be only one instance of Reader. If there are multiple readers, the second reader will get null pointer exception, as the message it want to read is already read by the first Reader. Kindly note that this is from my experience. Let me know, If I am wrong.