I am trying to visually cluster nodes in a network based on the weight of their edges: nodes with high edge weight should be close by in the 2d Euclidean space of the output image.
I built my network with NetworkX and converted it to PyVis in order to display it.
I'm wondering if there's a way to set up PyVis so that nodes with high weight edges between them are close by, ideally I would love it to look like this example.
I'm also wondering if instead I should pre-emptively cluster the nodes in NetworkX, tag them, and then use this information to display them close by (for clustering see this answer).
Another alternative would be to remove all edges except for those with high relative value (0.0075 in my example below), but I don't know if this would help with getting PyVis to visually cluster the nodes together.
Or are there any other alternatives to obtain something nice (PageRank, etc)? There seems to be a dearth of information on how to properly visualise graph data (I haven't found much apart from this).
Here's a MWE to reproduce what I'm seeing; basically, in this minimal example, I would like to see nodes [7,9,16,51,57] be clustered close together, compared to the other nodes (ideally I would love it to look like this example - I can always figure out the colouring later on):
def draw_graph3( networkx_graph, notebook=True, output_filename='graph.html', open_output=False,
show_buttons=False, only_physics_buttons=False ):
# import
from pyvis import network as net
# make a pyvis network
pyvis_graph = net.Network( notebook=notebook )
pyvis_graph.width = '1000px'
# for each node and its attributes in the networkx graph
for node, node_attrs in networkx_graph.nodes( data=True ):
pyvis_graph.add_node( node, **node_attrs )
# print(node,node_attrs)
# for each edge and its attributes in the networkx graph
for source, target, edge_attrs in networkx_graph.edges( data=True ):
# if value/width not specified directly, and weight is specified, set 'value' to 'weight'
if not 'value' in edge_attrs and not 'width' in edge_attrs and 'weight' in edge_attrs:
# place at key 'value' the weight of the edge
edge_attrs[ 'value' ] = edge_attrs[ 'weight' ]
# add the edge
pyvis_graph.add_edge( source, target, **edge_attrs )
# turn buttons on
if show_buttons:
if only_physics_buttons:
pyvis_graph.show_buttons( filter_=[ 'physics' ] )
else:
pyvis_graph.show_buttons()
# return and also save
pyvis_graph.show( output_filename )
if open_output:
import webbrowser
import os
webbrowser.open( f'file://{os.getcwd()}/{output_filename}' )
# Build NetworkX graph
import networkx as nx
pre_neurons = [7, 9, 16, 51, 57, 114, 122, 126, 129, 138, 170, 202, 208, 221, 254]
post_neurons = [7, 9, 16, 51, 57, 114, 122, 126, 129, 138, 170, 202, 208, 221, 254]
edges_pre = [7, 7, 7, 9, 9, 9, 9, 9, 16, 16, 16, 16, 16, 16, 16, 16, 16, 51, 51, 51, 51, 51, 51, 57, 57, 57, 57, 57, 57, 114, 114, 114, 114, 114, 114, 114, 114, 122, 122, 122, 126, 126, 126, 126, 126, 126, 129, 129, 129, 129, 129, 138, 138, 138, 138, 138, 138, 170, 170, 170, 170, 170, 170, 202, 202, 202, 202, 202, 202, 202, 208, 208, 208, 208, 208, 208, 221, 221, 254, 254, 254, 254, 254, 254, 254, 254]
edges_post = [122, 129, 170, 51, 122, 129, 138, 208, 7, 9, 51, 57, 114, 129, 170, 208, 221, 9, 129, 170, 202, 221, 254, 114, 129, 170, 202, 208, 254, 51, 122, 126, 129, 138, 202, 208, 254, 114, 202, 208, 9, 51, 122, 202, 221, 254, 51, 114, 126, 138, 170, 7, 57, 114, 126, 129, 170, 7, 9, 57, 126, 138, 221, 51, 114, 126, 129, 170, 208, 221, 7, 51, 138, 170, 202, 254, 51, 170, 9, 16, 57, 114, 122, 126, 138, 170]
edges_weights = [0.0, 0.0, 0.0, 0.0075, 0.0, 0.0, 0.0, 0.0, 0.0075, 0.0075, 0.0075, 0.0075, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0075, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0005, 0.0005, 0.0005, 0.0005, 0.0005, 0.0005, 0.0005, 0.0005, 0.0005, 0.0005, 0.0, 0.0, 0.0005, 0.0005, 0.0005, 0.0005, 0.0, 0.0005, 0.0005, 0.0005, 0.0005, 0.0, 0.0, 0.0005, 0.0005, 0.0005, 0.0005, 0.0, 0.0, 0.0, 0.0005, 0.0005, 0.0005, 0.0, 0.0005, 0.0005, 0.0005, 0.0005, 0.0005, 0.0005, 0.0, 0.0, 0.0005, 0.0005, 0.0005, 0.0005, 0.0, 0.0005, 0.0, 0.0, 0.0, 0.0005, 0.0005, 0.0005, 0.0005, 0.0005]
g = nx.Graph()
g.add_nodes_from( pre_neurons )
g.add_nodes_from( post_neurons )
for i, j, w in zip( edges_pre, edges_post, edges_weights ):
g.add_edge( i, j, weight=w )
a = draw_graph3( g, output_filename='test.html', open_output=True, notebook=False, show_buttons=True,
only_physics_buttons=True )
Related
I have an endpoint that returns me a file (pdf,xlsx) in the backend. When I send a request here with the get method, my expectation is that the file downloads directly to my computer.
Also, when I run this required request in the browser, the file downloads directly. I think the problem is caused by flutter.
I wonder if this has something to do with the Response Type?
Or what are your thoughts on this subject?
This is my Header of Dio
_headers() {
var authModel = read(cacheManagerProvider(BoxType.authBox))
.readModelFromBox<AuthModel>(BoxKey.authInfo);
var headerBody = {
'content-type': 'application/json',
'Accept': 'application/json',
};
if (authModel != null) {
headerBody.addAll(
{HttpHeaders.authorizationHeader: "Bearer ${authModel.access}"});
}
return headerBody;
}
I used the get method
Future<Response> get(
String apiUrl, {
ResponseType responseType = ResponseType.json,
Map<String, dynamic>? queryParameters,})async {
return await _dio.get(
apiUrl,
queryParameters: queryParameters,
options: Options(
responseType: responseType,
headers: _headers(),
),
);
}
Error returning in Debug console
Bad UTF-8 encoding (U+FFFD; REPLACEMENT CHARACTER) found while decoding string: ║ PK�[�TAMb��docProps/app.xmlM�=1D��q��A�Bb#�R��{/�dC�B~�9��noF�
g
. The Flutter team would greatly appreciate if you could file a bug explaining exactly what you were doing when this happened:
https://github.com/flutter/flutter/issues/new/choose
The source bytes were:
[226, 149, 145, 32, 80, 75, 3, 4, 20, 0, 0, 0, 8, 0, 239, 191, 189, 91, 239, 191, 189, 84, 7, 65, 77, 98, 239, 191, 189, 0, 0, 0, 239, 191, 189, 0, 0, 0, 16, 0, 0, 0, 100, 111, 99, 80, 114, 111, 112, 115, 47, 97, 112, 112, 46, 120, 109, 108, 77, 239, 191, 189, 61, 11, 2, 49, 16, 68, 239, 191, 189, 239, 191, 189, 113, 239, 191, 189, 239, 191, 189, 65, 239, 191, 189, 66, 98, 64, 239, 191, 189, 82, 239, 191, 189, 239, 191, 189, 15, 123, 27, 47, 239, 191, 189, 100, 67, 239, 191, 189, 66, 126, 239, 191, 189, 57, 239, 191, 189, 239, 191, 189, 110, 30, 111, 24, 70, 239, 191, 189, 10, 103, 10]
Solution: Change responseType from ResponseType.json to ResponseType.bytes
I'm trying to turn a string of a JWT token signature into a uint8array.
lX_aBSgGVYWd2FL6elRHoPJ2nab0IkmmX600cwZPCyK_SazZ-pzBUGDDQ0clthPVAtoS7roHg14xpEJlcSJUZBA7VTlPiDCOrkie_Hmulj765qS44t3kxAYduLhNQ-VN
The expected outcome should be the following with the size of 96 (base64url encoding)
[149, 127, 218, 5, 40, 6, 85, 133, 157, 216, 82, 250, 122, 84, 71, 160, 242, 118, 157, 166, 244, 34, 73, 166, 95, 173, 52, 115, 6, 79, 11, 34, 191, 73, 172, 217, 250, 156, 193, 80, 96, 195, 67, 71, 37, 182, 19, 213, 2, 218, 18, 238, 186, 7, 131, 94, 49, 164, 66, 101, 113, 34, 84, 100, 16, 59, 85, 57, 79, 136, 48, 142, 174, 72, 158, 252, 121, 174, 150, 62, 250, 230, 164, 184, 226, 221, 228, 196, 6, 29, 184, 184, 77, 67, 229, 77]
But I coudn't reverse engineer this I kept getting a size of 64 instead
[103, 10, 34, 6, 104, 125, 101, 234, 5, 23, 143, 198, 138, 228, 164, 134, 170, 116, 240, 159, 82, 223, 58, 73, 9, 49, 70, 51, 52, 229, 241, 5, 45, 146, 219, 135, 53, 177, 148, 181, 210, 164, 145, 59, 99, 95, 35, 46, 212, 62, 247, 142, 115, 234, 186, 88, 173, 148, 16, 157, 235, 29, 62, 93
Here's my function that was originally used for base16 and I'm now try to modify this to work for base64url
const convert = (str) => {
const segmented = str.match(/.{1,2}/g);
const arr = segmented.map((segment, i) => {
const powers = [16,1];
const alphabets = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789-_";
const two_bits = segment.split('');
const result_arr = two_bits.map((hex, i) => {
const decimal = alphabets.indexOf(hex);
const sum_of_power = powers[i] * decimal;
return sum_of_power;
});
const result = result.reduce((a, b) => a + b);
return result;
});
return new Uint8Array(arr);
};
Any help would be appreciated!
Here's the base64url decode code, taken from my jose universal javascript module which I suggest you use for any and all JOSE functionality instead of writing your own.
const decodeBase64 = (encoded) => {
return new Uint8Array(atob(encoded)
.split('')
.map((c) => c.charCodeAt(0)));
};
const decode = (input) => {
try {
return decodeBase64(input.replace(/-/g, '+').replace(/_/g, '/').replace(/\s/g, ''));
}
catch (_a) {
throw new TypeError('The input to be decoded is not correctly encoded.');
}
};
decode('lX_aBSgGVYWd2FL6elRHoPJ2nab0IkmmX600cwZPCyK_SazZ-pzBUGDDQ0clthPVAtoS7roHg14xpEJlcSJUZBA7VTlPiDCOrkie_Hmulj765qS44t3kxAYduLhNQ-VN')
Using the _localFile and readFile functions below I can choose the correct file and read it as bytes however I am unable to convert back to the correct text in the file. When using utf8 decode I get the following error
Unhandled Exception: FormatException: Unexpected extension byte (at offset 10).
The List of int I get from reading the file starts with
[80, 75, 3, 4, 20, 0, 8, 8, 8, 0, 135, 117, 116, 81, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 11, 0, 0, 0, 95, 114, 101, 108, 115, 47, 46, 114, 101, 108, 115, 173, 210, 193, 74, 3, 49, 16, 6, 224, 123, 159, 98, 153, 123, 119, 182, 85, 68, 100, 179, 189, 136, 208, 155, 72, 125, 128, 144, 204, 238, 6, 155, 76, 72, 166, 90, 223, 222, 80, 10, 186, 80, 86, 193, 30, 51, 249, 231, 231, 35, 164, 221, 28, 253, 190, 122, 167, 148, 29, 7, 5, 171, 186, 129, 138, 130, 97, 235, 194, 160, 224, 117, 247, 180, 188, 135, 77, 183, 104, 95, 104, 175, 165, 68, 242, 232, 98, 174, 202, 78, 200, 10, 70, 145, 248, 128, 152, 205, 72, 94, 231, 154, 35, 133, 114, 211, 115, 242, 90, 202, 49, 13, 24, 181, 121, 211, 3, 225, 186, 105, 238, 48, 253, 236, 128, 110, 210, 89, 109, 173, 130, 180, 181, 43, 168, 118, 159, 145, 254, 215, 141, 158, 68, 91, 45, 26, 13, 39, 90, 198, 84, 182, 147, 56, 202, 165, 92, 167, 129, 68, 129, 101, 243, 92, 198, 249, 148, 168, 75, 51, 224, 101, 208, 250, 239, 32, 238, 123, 103, 232, 145, 205, 193, 83, 144, 75, 46, 5...
and has a length of 24596 where as the text file only has 5601 characters. How do I get the contents of the file I have read in?
Thanks
Future<List<int>> readFile() async {
try {
final file = await _localFile;
// Read the file.
List<int> contents = await file.readAsBytes();
return contents;
} catch (e) {
print(e);
// If encountering an error, return 0.
return [0];
}
}
Future<File> get _localFile async {
final path = await FlutterDocumentPicker.openDocument();
return File('$path');
}
To convert the List<int> data correctly, you need to know how it was encoded. If you do not know, it may be useful to try to open the file with the Windows notepad and see which encoding offers you in the "Save as" window, and/or do some conversion tests with online tools (this is a practical approach, because it is not always possible to understand in retrospect which encoding was used). Then you can proceed with the decoding:
for UTF-8, you can use the built-in dart:convert package;
for UTF-16 and UTF-32, you can use the utf package (it's discontinued, but I didn't find alternatives);
for ANSI: ANSI characters after 128 change according to the country, so the exact charset ("code page") used must be identified. For example in Western Europe (for latin scripts), this can be Windows-1250 or Windows-1252. The only package I found that supports these encodings is charset_converter (it supports various charset, but it's available only for Android and iOS, because it relies on the native API of these systems, for example in Android on the API offered by Java).
Example for ANSI (which seems to be your case), for the "windows1250" code page:
(after including charset_converter in dependencies and importing it)
List<int> contents = await file.readAsBytes();
var decoded = await CharsetConverter.decode("windows1250", Uint8List.fromList(contents));
To read ANSI encoded files, the enough_convert package can read them and it works on all platforms! 😀
I am building a chess engine in Swift based off a tutorial written in Java. In the tutorial, Java's signed 64 bit integer long has a static method called reverse(long i) that "Returns the value obtained by reversing the order of the bits in the two's complement binary representation of the specified long value." It basically reverses the order of the binary bits so that, using 8-bit for simplicity's sake, 01000000 would become 00000010.
I am curious if there is a way to accomplish the same thing with Swift's UInt64. I understand that Java's long (signed, 2s complement) and Swift's UInt64 (unsigned) are different, but I imagine it wouldn't be hard to do this with UInt64.
I tried UInt64.byteSwapped, but this doesn't seem to be getting me the behavior I expect.
No, the Swift standard libraries do not provide a method to reverse the order of bits in an integer, see for example the discussion Bit reversal in the Swift forum.
One can use the C methods from Bit Twiddling Hacks, either by importing C code to Swift, or by translating it to Swift.
As an example, I have taken the loop-based variant
unsigned int s = sizeof(v) * CHAR_BIT; // bit size; must be power of 2
unsigned int mask = ~0;
while ((s >>= 1) > 0)
{
mask ^= (mask << s);
v = ((v >> s) & mask) | ((v << s) & ~mask);
}
because that is not restricted to a certain integer size. It can be translated to Swift as an extension to FixedWidthInteger so that it works with integers of all sizes:
extension FixedWidthInteger {
var bitSwapped: Self {
var v = self
var s = Self(v.bitWidth)
precondition(s.nonzeroBitCount == 1, "Bit width must be a power of two")
var mask = ~Self(0)
repeat {
s = s >> 1
mask ^= mask << s
v = ((v >> s) & mask) | ((v << s) & ~mask)
} while s > 1
return v
}
}
Examples:
print(String(UInt64(1).bitSwapped, radix: 16))
// 8000000000000000
print(String(UInt64(0x8070605004030201).bitSwapped, radix: 16))
// 8040c0200a060e01
print(String(UInt16(0x1234).bitSwapped, radix: 16))
// 2c48
Another option would be to use byteSwapped to reverse the order of bytes first, and then reverse the order of the bits in each byte with a (precomputed) lookup table:
fileprivate let bitReverseTable256: [UInt8] = [
0, 128, 64, 192, 32, 160, 96, 224, 16, 144, 80, 208, 48, 176, 112, 240,
8, 136, 72, 200, 40, 168, 104, 232, 24, 152, 88, 216, 56, 184, 120, 248,
4, 132, 68, 196, 36, 164, 100, 228, 20, 148, 84, 212, 52, 180, 116, 244,
12, 140, 76, 204, 44, 172, 108, 236, 28, 156, 92, 220, 60, 188, 124, 252,
2, 130, 66, 194, 34, 162, 98, 226, 18, 146, 82, 210, 50, 178, 114, 242,
10, 138, 74, 202, 42, 170, 106, 234, 26, 154, 90, 218, 58, 186, 122, 250,
6, 134, 70, 198, 38, 166, 102, 230, 22, 150, 86, 214, 54, 182, 118, 246,
14, 142, 78, 206, 46, 174, 110, 238, 30, 158, 94, 222, 62, 190, 126, 254,
1, 129, 65, 193, 33, 161, 97, 225, 17, 145, 81, 209, 49, 177, 113, 241,
9, 137, 73, 201, 41, 169, 105, 233, 25, 153, 89, 217, 57, 185, 121, 249,
5, 133, 69, 197, 37, 165, 101, 229, 21, 149, 85, 213, 53, 181, 117, 245,
13, 141, 77, 205, 45, 173, 109, 237, 29, 157, 93, 221, 61, 189, 125, 253,
3, 131, 67, 195, 35, 163, 99, 227, 19, 147, 83, 211, 51, 179, 115, 243,
11, 139, 75, 203, 43, 171, 107, 235, 27, 155, 91, 219, 59, 187, 123, 251,
7, 135, 71, 199, 39, 167, 103, 231, 23, 151, 87, 215, 55, 183, 119, 247,
15, 143, 79, 207, 47, 175, 111, 239, 31, 159, 95, 223, 63, 191, 127, 255]
extension FixedWidthInteger {
var bitSwapped: Self {
var value = self.byteSwapped
withUnsafeMutableBytes(of: &value) {
let bytes = $0.bindMemory(to: UInt8.self)
for i in 0..<bytes.count {
bytes[i] = bitReverseTable256[Int(bytes[i])]
}
}
return value
}
}
The following code streams a file to a process.
I want to stream audio/mp3
to many users who will hear it via html5 audio tag.
How can it be done via File.stream!?
defmodule Test do
def start do
p = spawn(Test, :say, [])
send p, {self, "a message"}
end
def say do
receive do
{from, msg} ->
IO.puts "Process #{inspect self} says: #{msg}"
stream_bytes = 128
File.stream!("./song.mp3", [], stream_bytes)
|> Enum.each(fn chunk ->
IO.inspect chunk
end)
say
end
end
end
$: iex test.ex
iex(1)> Test.start
output:
<<171, 46, 254, 26, 163, 32, 178, 27, 0, 75, 17, 35, 4, 236, 51, 57, 5, 144, 154, 198, 166, 47, 62, 4, 61, 85, 67, 135, 16, 34, 82, 49, 57, 176, 131, 96, 116, 152, 232, 24, 32, 140, 220, 67, 73, 128, 165, 178, 230, 202, ...>>
<<100, 220, 156, 191, 38, 0, 161, 117, 80, 16, 102, 91, 22, 5, 8, 66, 26, 7, 193, 155, 193, 66, 198, 28, 157, 244, 65, 131, 204, 240, 5, 172, 143, 44, 173, 85, 144, 2, 157, 144, 145, 97, 200, 236, 16, 49, 149, 150, 133, 67, ...>>
<<150, 54, 37, 254, 192, 218, 218, 26, 69, 231, 88, 124, 33, 129, 169, 66, 117, 52, 214, 134, 130, 103, 85, 130, 48, 6, 144, 221, 153, 132, 8, 181, 26, 27, 83, 140, 54, 117, 149, 7, 60, 144, 237, 248, 132, 12, 210, 51, 103, 116, ...>>
<<57, 2, 143, 220, 198, 182, 22, 177, 231, 126, 187, 147, 33, 9, 1, 5, 164, 2, 36, 105, 47, 255, 255, 255, 255, 255, 245, 54, 51, 225, 104, 98, 1, 184, 148, 206, 50, 135, 230, 28, 50, 47, 144, 134, 53, 16, 64, 130, 192, 198, ...>>
..............
how can I use JavaScript to read this binary data and hear it via audio tag ?
If you're using a plug based web framework it should be reasonably straight forward. This is possible if you're using plug directly or if you're using it from within phoenix (which is based on plug).
Maybe a plug like this would do the trick
defmodule Audio do
#chunk_size 128
def init(opts), do: opts
def song(conn, _opts) do
conn = conn
|> send_chunked(200)
|> put_resp_header("content-type", "audio/mpeg")
File.stream!("/some/song/somewhere.mp3", [], #chunk_size)
|> Enum.into(conn)
end
end
Maybe you want to hook up your plug to a phoenix router like this
defmodule MyApp.Router do
use MyApp.Web, :router
get "/the_song", Audio, :song
end
Then in your page
<audio src="/the_song">
Your browser does not support the <code>audio</code> element.
</audio>