Discussion:
Getting valid JSON output with xo(1)
Matt Churchyard
2018-09-08 13:41:10 UTC
Permalink
Hello,

Sorry if this is the wrong list, I wasn't sure which was the most
appropriate.

I've been asked to look at generating machine output from vm-bhyve using
xo(1), and though it would be an interesting task. However, I'm coming
across a few issues getting valid output. I've reduced my code down to the
following example -

xo -pX --open bhyve/machines
xo -pX --depth 2 --wrap machine "{:name},{:memory}" test1 2GB
xo -pX --depth 2 --wrap machine "{:name},{:memory}" test2 4GB
xo -pX --close bhyve/machines

This produces the following XML output which looks good -

<bhyve>
<machines>
<machine>
<name>test1</name>
<memory>2GB</memory>
</machine>
<machine>
<name>test2</name>
<memory>4GB</memory>
</machine>
</machine>
</bhyve>

If I choose HTML output I get the following invalid output which is missing
closing </div> tags.

<div class="line">
<div class="data" data-tag="name">test1</div>
<div class="text">,</div>
<div class="data" data-tag="memory">2GB</div>
<div class="line">
<div class="data" data-tag="name">test2</div>
<div class="text">,</div>
<div class="data" data-tag="memory">4GB</div>

This can be fixed by adding "\n" to the end of the format lines, which may
be required, but it seems strange that I can so easily generate broken
output. I'm not too bothered about HTML though as it seems fairly useless,
JSON is my biggest issue -

With JSON specified I get the following output, which has 3 major issues:

"bhyve": {
"machines": {
"machine": {
"name": "test1",
"memory": "2GB"
} "machine": {
"name": "test2",
"memory": "4GB"
}
}
}

1) The entire document needs to be inside braces "{}" so that "bhyve" is a
member of an object.
2) There's a missing comma between the two machine entries
3) "machines" is an object, which erroneously contains 2 "machine" keys

(There's also a 4th issue that the second "machine" entry really should be
on the line below, but that's just a minor visual thing)

Now 1 & 2 can be fixed fairly easily by just outputting the brace and comma
characters myself. (I do also appreciate that in order to output the commas
in the correct place, xo would need to either get all the records in one
go, or be told which are part of a list and which are first or last).
However it seems to me that xo should handle being given the exact same
input, and produce valid HTML/JSON or XML just by the user specifying their
output choice. I shouldn't really have to inject additional characters
myself or alter the input for certain formats in order to get valid output.

The 3rd issue is a bit more awkward. The xo(1) man page is fairly short and
I can't see any obvious way that I can get it to produce a basic array
(square brackets - []) rather than an object, meaning that it doesn't seem
possible to create a list of objects. This works with XML due to a list
being made up by effectively just repeating the same element. If I can't
create a list with JSON, it seems to severely limit the usefulness of the
JSON option (Which unfortunately is the most desired format).

I could possibly use the machine name as the object key, but this isn't
ideal, and again means I'll need to start running different xo commands
based on the desired output format.

I don't know if libxo can produce JSON arrays if you're actually using the
proper API, but these problems (If I'm not already missing something
obvious) could probably be fixed fairly easily if there was some option to
xo to specify when it should create a list, and whether the current call is
the first entry or not.

Regards,
Matt
Richard Perini
2018-09-09 01:34:58 UTC
Permalink
Post by Matt Churchyard
Hello,
Sorry if this is the wrong list, I wasn't sure which was the most
appropriate.
I've been asked to look at generating machine output from vm-bhyve using
xo(1), and though it would be an interesting task. However, I'm coming
across a few issues getting valid output. I've reduced my code down to the
following example -
[ ... ]

I've never used xo but have used "jo" to perform similar tasks.

https://github.com/jpmens/jo

Its JSON only. I don't think there's a port for it, but it builds simply
on FreeBSD-10 through FreeBSD-12.

Richard Perini
Ramico Australia Pty Ltd Sydney, Australia ***@ci.com.au +61 2 9552 5500
-----------------------------------------------------------------------------
"The difference between theory and practice is that in theory there is no
difference, but in practice there is"
Matt Churchyard
2018-09-09 14:10:09 UTC
Permalink
Hi Stefan,

I did include the commands used to generate the sample output in my first
message. I am using the xo(1) utility, not libxo directly. I did mention
this might not be the right list (you've obviously assumed I'm directly
using libxo), but it does seem to me this is more an issue with the xo
utility, rather than just a usage error.

I've since looked further into the libxo documentation, and as you suggest,
it does seem that I need to be using the xo_open_list/xo_close_list
functions to generate the right output. However, these are not exposed via
xo(1). The documentation for xo_open_list describes how it is used to
create lists of individual entities, whereas the "l" modifier (which is all
I can access via xo(1)) creates "leaf lists", which are only for single
values. As such, it seems there's a big hole in the functionality of xo(1),
meaning it can only really be used to create very simple JSON documents.

As I mentioned in my first message, it looks like xo(1) could do with some
way to open/close lists, similar to the --open/--close options, and a way
to specify that a single call should be output as a part of a list.

Matt
Post by Matt Churchyard
Post by Matt Churchyard
Hello,
Sorry if this is the wrong list, I wasn't sure which was the most
appropriate.
I've been asked to look at generating machine output from vm-bhyve using
xo(1), and though it would be an interesting task. However, I'm coming
across a few issues getting valid output. I've reduced my code down to
the
Post by Matt Churchyard
following example -
xo -pX --open bhyve/machines
xo -pX --depth 2 --wrap machine "{:name},{:memory}" test1 2GB
xo -pX --depth 2 --wrap machine "{:name},{:memory}" test2 4GB
xo -pX --close bhyve/machines
This produces the following XML output which looks good -
<bhyve>
<machines>
<machine>
<name>test1</name>
<memory>2GB</memory>
</machine>
<machine>
<name>test2</name>
<memory>4GB</memory>
</machine>
</machine>
</bhyve>
If I choose HTML output I get the following invalid output which is
missing
Post by Matt Churchyard
closing </div> tags.
<div class="line">
<div class="data" data-tag="name">test1</div>
<div class="text">,</div>
<div class="data" data-tag="memory">2GB</div>
<div class="line">
<div class="data" data-tag="name">test2</div>
<div class="text">,</div>
<div class="data" data-tag="memory">4GB</div>
This can be fixed by adding "\n" to the end of the format lines, which
may
Post by Matt Churchyard
be required, but it seems strange that I can so easily generate broken
output. I'm not too bothered about HTML though as it seems fairly
useless,
Post by Matt Churchyard
JSON is my biggest issue -
It would be a lot easier to help if you posted a (stripped down) version of
the code used to generate this output.
You are not correctly using the list functions, and thus the outer lists
are
not closed. I had suggested a better syntax and semantics for these
functions
to automatically take care of such cases, but they were rejected by the
main
libxo author.
Every xo_open_list() must be matched by a matching xo_close_list() and you
may need to use another type (list, container, instance) than you currently
do.
Did you run xolint to test your usage of xo functions?
Regards, STefan
Stefan Esser
2018-09-09 16:13:04 UTC
Permalink
Post by Matt Churchyard
Hi Stefan,
I did include the commands used to generate the sample output in my first
message. I am using the xo(1) utility, not libxo directly. I did mention
this might not be the right list (you've obviously assumed I'm directly
using libxo), but it does seem to me this is more an issue with the xo
utility, rather than just a usage error.
Hi Matt,

I'm sorry - you are of course right and there was everything required to
reproduce the effect in your previous mail ...
Post by Matt Churchyard
I've since looked further into the libxo documentation, and as you suggest,
it does seem that I need to be using the xo_open_list/xo_close_list
functions to generate the right output. However, these are not exposed via
xo(1). The documentation for xo_open_list describes how it is used to
create lists of individual entities, whereas the "l" modifier (which is all
I can access via xo(1)) creates "leaf lists", which are only for single
values. As such, it seems there's a big hole in the functionality of xo(1),
meaning it can only really be used to create very simple JSON documents.
The following code fragment in libxo.c does suppress the terminating new-line
character on purpose if depth != 0:

case XO_STYLE_JSON:
pre_nl = XOF_ISSET(xop, XOF_PRETTY) ? "\n" : "";
ppn = (xop->xo_depth <= 1) ? "\n" : "";

xo_depth_change(xop, name, -1, -1, XSS_CLOSE_CONTAINER, 0);
rc = xo_printf(xop, "%s%*s}%s", pre_nl, xo_indent(xop), "", ppn);
xop->xo_stack[xop->xo_depth].xs_flags |= XSF_NOT_FIRST;
break;

The reason appears to be, that closing the outer level construct will print
a new-line before other text. If you omit the second "xo --wrap", everything
will look OK. And if you test with --depth 0, you'll see that the output is
actually terminated with a new-line.
Post by Matt Churchyard
As I mentioned in my first message, it looks like xo(1) could do with some
way to open/close lists, similar to the --open/--close options, and a way
to specify that a single call should be output as a part of a list.
I do not see a better solution than adding an "echo" command between calls
to "xo --wrap", as you already mentioned as a work-around. This is simpler
than adding list_open/close functions and it has the same effect, if you
know that you are emitting JSON.

Best regards, STefan
Matt Churchyard
2018-09-10 19:34:37 UTC
Permalink
Thanks Stefan

For now I’ve altered my code to output the extra characters when needed,
and switch {} to [] using tr.

I may raise an issue with the libxo guys and see what happens, as it seems
to be a bit of a pain that I have to go through all this to generate a
fairly simple JSON document.

Matt
Post by Stefan Esser
Post by Matt Churchyard
Hi Stefan,
I did include the commands used to generate the sample output in my first
message. I am using the xo(1) utility, not libxo directly. I did mention
this might not be the right list (you've obviously assumed I'm directly
using libxo), but it does seem to me this is more an issue with the xo
utility, rather than just a usage error.
Hi Matt,
I'm sorry - you are of course right and there was everything required to
reproduce the effect in your previous mail ...
Post by Matt Churchyard
I've since looked further into the libxo documentation, and as you
suggest,
Post by Matt Churchyard
it does seem that I need to be using the xo_open_list/xo_close_list
functions to generate the right output. However, these are not exposed
via
Post by Matt Churchyard
xo(1). The documentation for xo_open_list describes how it is used to
create lists of individual entities, whereas the "l" modifier (which is
all
Post by Matt Churchyard
I can access via xo(1)) creates "leaf lists", which are only for single
values. As such, it seems there's a big hole in the functionality of
xo(1),
Post by Matt Churchyard
meaning it can only really be used to create very simple JSON documents.
The following code fragment in libxo.c does suppress the terminating new-line
pre_nl = XOF_ISSET(xop, XOF_PRETTY) ? "\n" : "";
ppn = (xop->xo_depth <= 1) ? "\n" : "";
xo_depth_change(xop, name, -1, -1, XSS_CLOSE_CONTAINER, 0);
rc = xo_printf(xop, "%s%*s}%s", pre_nl, xo_indent(xop), "", ppn);
xop->xo_stack[xop->xo_depth].xs_flags |= XSF_NOT_FIRST;
break;
The reason appears to be, that closing the outer level construct will print
a new-line before other text. If you omit the second "xo --wrap", everything
will look OK. And if you test with --depth 0, you'll see that the output is
actually terminated with a new-line.
Post by Matt Churchyard
As I mentioned in my first message, it looks like xo(1) could do with
some
Post by Matt Churchyard
way to open/close lists, similar to the --open/--close options, and a way
to specify that a single call should be output as a part of a list.
I do not see a better solution than adding an "echo" command between calls
to "xo --wrap", as you already mentioned as a work-around. This is simpler
than adding list_open/close functions and it has the same effect, if you
know that you are emitting JSON.
Best regards, STefan
Phil Shafer
2018-09-11 01:29:16 UTC
Permalink
Post by Matt Churchyard
Sorry if this is the wrong list, I wasn't sure which was the most
appropriate.
Best path for xo-related bugs is https://github.com/Juniper/libxo/issues
Post by Matt Churchyard
If I choose HTML output I get the following invalid output which is missing
closing </div> tags.
<div class="line">
<div class="data" data-tag="name">test1</div>
<div class="text">,</div>
<div class="data" data-tag="memory">2GB</div>
<div class="line">
<div class="data" data-tag="name">test2</div>
<div class="text">,</div>
<div class="data" data-tag="memory">4GB</div>
This can be fixed by adding "\n" to the end of the format lines, which may
be required, but it seems strange that I can so easily generate broken
output.
Yes, without a trailing newline, libxo can't declare the output
line "over", so it doesn't make the </div>. The problem is that
there's no mechanism to "xo" that your output line is continued
from the previous one, and since "xo" is stateless, it will happily
start the next chunk of output with a new open div. I'll add a
"--continuation" option. (#58)
Post by Matt Churchyard
"bhyve": {
"machines": {
"machine": {
"name": "test1",
"memory": "2GB"
} "machine": {
"name": "test2",
"memory": "4GB"
}
}
}
There's an "xo" option called "--not-first" for exactly these
situations; place that option on the second invocation so it will
know to emit the leading comma. This should be documented (#59).
Post by Matt Churchyard
1) The entire document needs to be inside braces "{}" so that "bhyve" is a
member of an object.
"xo" doesn't currently do this, since it assume you're generating
JSON piecemeal and have a better idea that it of what you want
to do with the output, but I've opened issue #60 for adding
a "--object-wrapper" option that will perform this.
Post by Matt Churchyard
2) There's a missing comma between the two machine entries
The "--not-first" option will repair this.
Post by Matt Churchyard
3) "machines" is an object, which erroneously contains 2 "machine" keys
This is a bug. "xo" lacks a means of knowing that an object is a
list. One might expect that labeling "name as a key would perform
this, but it doesn't, and it's not clear that this is enough
information. Since "xo" is stateless, it cannot really handle
that "--wrapper machine" applies only to the first item. It
would have to know not to generate the closing "}" for machine,
but it's stateless, so I'd need a means of telling it that.

In C code, we have "xo_open_list" and "xo_open_instance", like:

https://libxo.readthedocs.io/en/latest/api.html?highlight=xo_open_instance#lists-and-instances

But with "xo" I don't have lists and instances. I'll need to
find a means of adding these, though that would make "xo" more
cumbersome and it would lack the FSM (and the stack) that catches
missing calls in libxo.

xo $opts --open-list machine
for name in red green blue; do
xo $opts --open-instance machine
xo $opts "Machine {k:name} has {:memory}\n" $name `get-mem ~name`
xo $opts --close-instance machine
done
xo $opts --close-list machine

Which is very exact but less that beautiful. This is #61.

Thanks,
Phil
Matt Churchyard
2018-09-11 10:44:15 UTC
Permalink
Hi Phil,

Thanks for the detailed response.

The part about HTML makes perfect sense. Each outer div is a "line", so
without the "\n", you may well want to create further output on the same
line by making additional calls. (Assuming of course that you can use the
--not-first option to stop it creating another "line" div)

As you mention, the fact that xo(1) is designed to be called multiple times
in order to build up the output from fragments makes some of this stuff
awkward. Thanks for pointing out the --not-first option.

I think the sample from #61 looks fairly reasonable. Not that I want to
create you guys extra work..., I've no doubt your primary focus is libxo
itself which already works fine. Just seems that the xo command line
utility is quite hampered if it can't create lists (in JSON at least). At
the moment I'm getting around it fairly simply by using something like the
following, (the tr calls fortunately only have an effect on JSON output.),
which works, but is a lot of jumping through hoops just to get valid JSON.

xo -pJ --open machines | tr '{' '['
(if JSON & not first) echo ","
(if JSON) echo "{"
(if JSON) xo -pJ "{:key1}{:key2}{:key3}" data1 data2 data3
(if not JSON) xo -pJ --wrap machine "{:key1}{:key2}{:key3}" data1 data2
data3
(if JSON) echo "}"
xo -pJ --close machines | tr '}' ']'

Regards,
Matt
Post by Phil Shafer
Post by Matt Churchyard
Sorry if this is the wrong list, I wasn't sure which was the most
appropriate.
Best path for xo-related bugs is https://github.com/Juniper/libxo/issues
Post by Matt Churchyard
If I choose HTML output I get the following invalid output which is
missing
Post by Matt Churchyard
closing </div> tags.
<div class="line">
<div class="data" data-tag="name">test1</div>
<div class="text">,</div>
<div class="data" data-tag="memory">2GB</div>
<div class="line">
<div class="data" data-tag="name">test2</div>
<div class="text">,</div>
<div class="data" data-tag="memory">4GB</div>
This can be fixed by adding "\n" to the end of the format lines, which may
be required, but it seems strange that I can so easily generate broken
output.
Yes, without a trailing newline, libxo can't declare the output
line "over", so it doesn't make the </div>. The problem is that
there's no mechanism to "xo" that your output line is continued
from the previous one, and since "xo" is stateless, it will happily
start the next chunk of output with a new open div. I'll add a
"--continuation" option. (#58)
Post by Matt Churchyard
"bhyve": {
"machines": {
"machine": {
"name": "test1",
"memory": "2GB"
} "machine": {
"name": "test2",
"memory": "4GB"
}
}
}
There's an "xo" option called "--not-first" for exactly these
situations; place that option on the second invocation so it will
know to emit the leading comma. This should be documented (#59).
Post by Matt Churchyard
1) The entire document needs to be inside braces "{}" so that "bhyve" is a
member of an object.
"xo" doesn't currently do this, since it assume you're generating
JSON piecemeal and have a better idea that it of what you want
to do with the output, but I've opened issue #60 for adding
a "--object-wrapper" option that will perform this.
Post by Matt Churchyard
2) There's a missing comma between the two machine entries
The "--not-first" option will repair this.
Post by Matt Churchyard
3) "machines" is an object, which erroneously contains 2 "machine" keys
This is a bug. "xo" lacks a means of knowing that an object is a
list. One might expect that labeling "name as a key would perform
this, but it doesn't, and it's not clear that this is enough
information. Since "xo" is stateless, it cannot really handle
that "--wrapper machine" applies only to the first item. It
would have to know not to generate the closing "}" for machine,
but it's stateless, so I'd need a means of telling it that.
https://libxo.readthedocs.io/en/latest/api.html?highlight=xo_open_instance#lists-and-instances
But with "xo" I don't have lists and instances. I'll need to
find a means of adding these, though that would make "xo" more
cumbersome and it would lack the FSM (and the stack) that catches
missing calls in libxo.
xo $opts --open-list machine
for name in red green blue; do
xo $opts --open-instance machine
xo $opts "Machine {k:name} has {:memory}\n" $name `get-mem ~name`
xo $opts --close-instance machine
done
xo $opts --close-list machine
Which is very exact but less that beautiful. This is #61.
Thanks,
Phil
Loading...