I am trying to create a Common Lisp data structure to output some nested JSON like so:
{
"array_of_objects": {
"item1": {
"foo": {
"value": "x"
}
},
"item2": {
"bar": {
"value": "y"
}
}
},
"object": {
"lorem": {
"foo": "bar"
}
}
}
I've chosen the YASON library in order to do this, and started with this example:
(yason:with-output (*standard-output*)
(yason:with-object ()
(yason:encode-object-element "hello" "hu hu")
(yason:with-object-element ("harr")
(yason:with-array ()
(dotimes (i 3)
(yason:encode-array-element i))))))
However, I'm not familiar enough with lisp and lisp hash map data structures to truly understand how this works.
What's the most "lispy" way to represent the JSON posted and then output it to stdout. Happy to use any library needed if its quicklisp compatible.
What do you start with? If you want a data structure that will output this JSON, you need hash-tables inside hash-tables. Here's the Cookbook: https://lispcookbook.github.io/cl-cookbook/data-structures.html#hash-table
Below I use serapeum's dict, a literal way to represent hash-tables, and more concise in our case:
(defparameter ht (dict "array_of_objects"
(dict "item1"
(dict "foo"
(dict "value" "x")))))
and I create the JSON with jonathan:
(jojo:to-json ht)
"{\"array_of_objects\":{\"item1\":{\"foo\":{\"value\":\"x\"}}}}"
Related
Given this input in a file called plants.json
{
"flower": { "rose": 1 },
"tree": { "spruce": 1, "oak": 2, "oaky": 3 }
}
Filter it with zq so that it looks like this:
{
"flower": { "rose": 1 },
"tree": { "oak": 2, "oaky": 3 }
}
Effectively filtering a known and named nested object at the key tree to match oak.
Use zq's put operator (the :=) to redefine tree as a filtered search.
zq -f json 'tree:=unflatten((over tree | oak))' plants.json
{"flower":{"rose":1},"tree":{"oak":2,"oaky":3}}
This does a simple search on keys and values. If you want to match more precisely, you can use other functions where | oak is.
For example, if you have a document that also includes "soak":4
{
"flower": { "rose": 1 },
"tree": { "spruce": 1, "oak": 2, "oaky": 3, "soak": 4 }
}
Then this won't work because you'll get oak and oaky and soak. So use grep() and a regex for this case. This example demonstrates how to add the function grep to our previous command.
zq -f json 'tree:=unflatten((over tree | grep(/^oak/)))' plants.json
{"flower":{"rose":1},"tree":{"oak":2,"oaky":3}}
Note that tree:=unflatten(...) is the same as saying put tree:=unflatten(...) because put is an implied operator like oak is search oak. Some of the links on zq's doc site are currently broken but it is currently in the 2.6 language overview section.
After I saved a json file in Visual Studio Code, it automatically formats the document, that arrays are displayed vertically (by adding line breaks). Since I have giant integer arrays, it makes a quick look into the file nearly impossible. Example:
{
"big_integer_array": [
12,
15,
13,
1,
5,
8,
15,
14,
12,
...
],
"this_value_is_in_line_million": true
}
How can I tell the program, to display numeric arrays horizontally like this:
{
"big_integer_array": [12,15,13,1,5,8,15,14,12, ...],
"this_value_is_in_line_three": true
}
Solution
Use the extension esbenp.prettier-vscode
Set the prettier.printwidth to 99999999999
There is a new setting built-in to vscode v1.70 specifically for this situation:
JSON > Format: Keep Lines <= enable this in your Settings
"json.format.keepLines": true <= or use that setting in your settings.json
This feature allows one to keep the initial line positions upon line
formatting of JSON documents. For example an array containing values
on one line will still contain these values on the same line after
formatting. To enable this feature, enable the option JSON > Format: Keep Lines in the settings.
from https://github.com/microsoft/vscode-docs/blob/vnext/release-notes/v1_70.md#keep-newlines-when-formatting-json
Demo using the Insiders Build - first without the setting enabled and then after enabling the setting:
There is currently no way to customize the VS Code native formatter to format arrays in the way you suggest, however, the prettier formatter will configure them as you suggest. Prettier is a widely used formatter, and is used by a good majority of VS Code users.
The ID for the official prettier extension is: esbenp.prettier-vscode
It's important to note that Prettier will take care of arrays that have a single type of input for you. So if an array is all numbers, or all strings, however; if an array consist of mixed types, numbers, objects, arrays in arrays, strings, boolean values, etc...
Then the way prettier formats the array is contingent on how you place the brackets initially.
Formatting Mixed arrays with Prettier
For this example's sake, lets say have the following array in a json file.
{
"obj": {
"Array": [
"Apple",
"Broccoli",
"Coconut",
"Orange",
"Carrot",
{
"foo": "apple"
}
]
}
}
If you change the brackets in the array, so that the array looks like the following:
{
"obj": {
"Array": [
"Apple",
"Broccoli",
"Coconut",
"Orange",
"Carrot",
{"foo": "apple"}
]
}
}
Your basically telling prettier that you don't want to break up your embedded objects and arrays vertically, but rather, you want to keep them horizontally. Formatting the example above (using prettier) will result in your file looking like the example below:
{
"obj": {
"Array": ["Apple", "Broccoli", "Coconut", "Orange", "Carrot", { "foo": "apple" }]
}
}
However, if you format your array like this:
{
"obj": {
"Array": ["Apple", "Broccoli", "Coconut", "Orange", "Carrot", {
"foo": "apple"
}]
}
Then when you format the above example using prettier, you will produce the output below:
{
"obj": {
"Array": [
"Apple",
"Broccoli",
"Coconut",
"Orange",
"Carrot",
{
"foo": "apple"
}
]
}
}
Also note than the following settings in your VS Code's settings.json file can affect how prettier formats JSON as well:
"prettier.useTabs": true|false (says to use tab, or spaces)
"prettier.tabWidth": Numeric Value (Sets tab spacing qty)
"prettier.printWidth": Numeric Value (Sets line length)
"prettier.bracketSpacing": True|False (Adds/Removes spacing in brackets)
I have used mongo import to import data into mongodb from csv files. I am trying to retrieve data from an Mongodb realm service. The returned data for the entry is as follows:
{
"_id": "6124edd04543fb222e",
"Field1": "some string",
"Field2": {
"$numberDouble": "145.81"
},
"Field3": {
"$numberInt": "0"
},
"Field4": {
"$numberInt": "15"
},
"Field5": {
"$numberInt": "0"
}
How do I convert this into normal JSON by removing $numberInt and $numberDouble like :
{
"_id": "6124edd04543fb222e",
"Field1": "some string",
"Field2": 145.8,
"Field3": 0,
"Field4": 15,
"Field5": 0
}
The fields are also different for different documents so cannot use Mongoose directly. Are there any solutions to this?
Also would help to know why the numbers are being stored as $numberInt:"".
Edit:
For anyone with the same problem this is how I solved it.
The array of documents is in EJSON format instead of JSON like said in the upvoted answer. To covert it back into normal JSON, I used JSON.stringify to first convert each document I got from map function into string and then parsed it using EJSON.parse with
{strict:false} (this option is important)
option to convert it into normal JSON.
{restaurants.map((restaurant) => {
restaurant=EJSON.parse(JSON.stringify(restaurant),{strict:false});
}
EJSON.parse documentation here. The module to be installed and imported is mongodb-extjson.
The format with $numberInt etc. is called (MongoDB) Extended JSON.
You are getting it on the output side either because this is how you inserted your data (meaning your inserted data was incorrect, you need to fix the ingestion side) or because you requested extended JSON serialization.
If the data in the database is correct, and you want non-extended JSON output, you generally need to write your own serializers to JSON since there are multiple possibilities of how to format the data. MongoDB's JSON output format is the Extended JSON you're seeing in your first quote.
I am having the genericRecord as well as the schema. I want to parse all the fields and append some text only to those values whose type is String.
Assume that schema is like this:
{
"type":"record",
"name":"MyRecord",
"fields":[
{
"name":"a",
"type":[
"null",
"int",
{
"type":"map",
"values":"string"
}
]
}
]
}
I don't want to write nested if else statements that I have already done and recursively called them. Is there any other efficient way ?
I have a document like this:
{
"whoKnows" : {
"name" : "Jeff",
"phone" : "123-123-1234"
},
"anotherElement" : {
"name" : "Jeff",
"phone" : "321-321-3211"
}
}
How can any instance of "name" by queried? For example, using a wildcard may look something like,
db.collection.find( { "*.name" : "Jeff" } )
Or if regex was support in the element place, it might look like,
db.collection.find( { /.*\.name/ : "Jeff" } )
Is it possible to accomplish this using MongoDB?
Side note: I'm not looking for a solution like,
db.collection.find({
"$or": [
{ "whoKnows.name" : "Jeff" },
{ "anotherElement.name" : "Jeff" }
]
})
I need a truly relative path solution as I do not know what the parent element will be (unless there is a way to generate the name of every element - then I could dynamically generate the $or clause at runtime).
Everything about this is fairly horrible, you cannot possibly index on something like the "name" values and your "path" to each attribute is going to vary everywhere. So this is really bad for queries.
I notice you mention "nested" structures, and you still could accommodate this with a similar proposal and some additional tagging, but I want you to consider this "phone book" type example:
{
"phones": [
{
"type": "Home",
"name" : "Jeff",
"phone" : "123-123-1234"
},
{
"type": "Work",
"name" : "Jeff",
"phone" : "123-123-1234"
},
]
}
Since this is actually sub-documents within an array, fields like "name" always share the same path, so not only can you index these (which is going to be good for performance) but the query is very basic:
db.collection({ "phones.name": "Jeff" })
That does exactly what you need by finding "Jeff" in any "name" entry. If you need a hierachy, then add some fields in those sub-documents to indicate the parent/child relationship that you can use in post processing. Or even as a materialized path which could aid your queries.
It really is the better approach.
If you really must keep this kind of structure then at least do something like this with the JavaScript that will bail out on the first match at depth:
db.collection.find(
function () {
var found = false;
var finder = function( obj, field, value ) {
if ( obj.hasOwnProperty(field) && obj[field] == value )
found = true;
if (found) return true;
for( var n in obj ) {
if ( Object.prototype.toString.call(obj[n]) === "[object Object]" ) {
finder( obj[n], field, value );
if (found) return true;
}
}
};
finder( this, "name", "Jeff" );
return found;
}
)
The format there is shorthand notation for the $where operator, which is pretty bad news for performance, but your structure isn't offering much other choice. At any rate, the function should recurse into each nested document until the "field" with the "value" is found.
For anything of production scale, really look at changing the structure to something that can be indexed and accessed quickly. The first example should give you a starting point. Relying on arbitrary JavaScript for queries as your present structure constrains you to is bad news.
If these are similar instance, what stops you in putting these in an array? That would be easier to query.
In it's current form this looks as good as writing your own $where condition to parse all document structure and is not an efficient operation!
Although highly inefficient and I wouldn't suggest using this in a production environment, following is one of the simplest way (with its own various catches) you can query:
db.query.find({$where: function() { x = tojsononeline(this); return x.indexOf('"name" : "Jeff",') >= 0; } })
Please note that this will cause a tablescan and if you have a pre-condition you may want to specify that before the where clause in the query.