Discussion:
Transform a single .avdl or .avpr into one or many .avsc
Bertrand Dechoux
2013-05-17 12:46:35 UTC
Permalink
Hi,

I have lots of avro schemas and most of them are about complex objects (ie
nested definition of stuff). The syntax of avro idl is attractive in order
to build something more readable and thus maintainable. However, it looks
like I can't generate any avsc from a avdl (or the avpr generated from the
avdl). I understand what is a protocol and I don't need one but the idl
syntax is really attractive. Is there really no way to use it for that
purpose?

Regards

Bertrand


PS : I remember seeing a discussion about that subject but I can't find it.
Jeremy Kahn
2013-05-17 14:10:24 UTC
Permalink
The "types" field in a protocol (.avro) may get you what you need. The
corresponding schema objects should be able to render to well-formed avsc
objects.
Post by Bertrand Dechoux
Hi,
I have lots of avro schemas and most of them are about complex objects (ie
nested definition of stuff). The syntax of avro idl is attractive in order
to build something more readable and thus maintainable. However, it looks
like I can't generate any avsc from a avdl (or the avpr generated from the
avdl). I understand what is a protocol and I don't need one but the idl
syntax is really attractive. Is there really no way to use it for that
purpose?
Regards
Bertrand
PS : I remember seeing a discussion about that subject but I can't find it.
Doug Cutting
2013-05-17 19:11:08 UTC
Permalink
There's not a tool that does this currently. Note however that
existing tools will generate Java classes for each type in an IDL
file, so if you're only using Java then you might not need a .avsc
file for each type in the IDL.

It would not be hard to add a tool (or an option to an existing tool)
that wrote a .avsc file for each type in an .avdl file. One could
also add Maven support for this. If this is of interest, please file
an issue in Jira.

https://issues.apache.org/jira/browse/AVRO

If you're willing and able, please provide an implementation.
Otherwise hopefully someone else will help out.

Cheers,

Doug
Post by Jeremy Kahn
The "types" field in a protocol (.avro) may get you what you need. The
corresponding schema objects should be able to render to well-formed avsc
objects.
Post by Bertrand Dechoux
Hi,
I have lots of avro schemas and most of them are about complex objects (ie
nested definition of stuff). The syntax of avro idl is attractive in order
to build something more readable and thus maintainable. However, it looks
like I can't generate any avsc from a avdl (or the avpr generated from the
avdl). I understand what is a protocol and I don't need one but the idl
syntax is really attractive. Is there really no way to use it for that
purpose?
Regards
Bertrand
PS : I remember seeing a discussion about that subject but I can't find it.
Bertrand Dechoux
2013-05-21 19:50:37 UTC
Permalink
After browsing the (nice) API, it seems indeed trivial.

The Idl class allows to read/parse the related file.
http://avro.apache.org/docs/current/api/java/org/apache/avro/compiler/idl/Idl.html#Idl%28java.io.File%29

The Protocol object can then be requested from it.
http://avro.apache.org/docs/current/api/java/org/apache/avro/compiler/idl/Idl.html#ProtocolDeclaration%28%29

Of course, the types can then be requested from the Protocol itself.
http://avro.apache.org/docs/current/api/java/org/apache/avro/Protocol.html#getTypes%28%29

And then it is only a matter of serializing them. And actually the API
provides even a 'pretty' option.
http://avro.apache.org/docs/current/api/java/org/apache/avro/Schema.html#toString%28boolean%29

I will definitely look at it (ie with javac). Contributing it as a tool
would be nice. But I won't go into the maven land.

Regards

Bertrand
Post by Doug Cutting
There's not a tool that does this currently. Note however that
existing tools will generate Java classes for each type in an IDL
file, so if you're only using Java then you might not need a .avsc
file for each type in the IDL.
It would not be hard to add a tool (or an option to an existing tool)
that wrote a .avsc file for each type in an .avdl file. One could
also add Maven support for this. If this is of interest, please file
an issue in Jira.
https://issues.apache.org/jira/browse/AVRO
If you're willing and able, please provide an implementation.
Otherwise hopefully someone else will help out.
Cheers,
Doug
Post by Jeremy Kahn
The "types" field in a protocol (.avro) may get you what you need. The
corresponding schema objects should be able to render to well-formed avsc
objects.
Post by Bertrand Dechoux
Hi,
I have lots of avro schemas and most of them are about complex objects
(ie
Post by Jeremy Kahn
Post by Bertrand Dechoux
nested definition of stuff). The syntax of avro idl is attractive in
order
Post by Jeremy Kahn
Post by Bertrand Dechoux
to build something more readable and thus maintainable. However, it
looks
Post by Jeremy Kahn
Post by Bertrand Dechoux
like I can't generate any avsc from a avdl (or the avpr generated from
the
Post by Jeremy Kahn
Post by Bertrand Dechoux
avdl). I understand what is a protocol and I don't need one but the idl
syntax is really attractive. Is there really no way to use it for that
purpose?
Regards
Bertrand
PS : I remember seeing a discussion about that subject but I can't find it.
--
Bertrand Dechoux
Bertrand Dechoux
2013-05-21 21:44:50 UTC
Permalink
Waiting for a review : https://issues.apache.org/jira/browse/AVRO-1337

Regards

Bertrand
Post by Bertrand Dechoux
After browsing the (nice) API, it seems indeed trivial.
The Idl class allows to read/parse the related file.
http://avro.apache.org/docs/current/api/java/org/apache/avro/compiler/idl/Idl.html#Idl%28java.io.File%29
The Protocol object can then be requested from it.
http://avro.apache.org/docs/current/api/java/org/apache/avro/compiler/idl/Idl.html#ProtocolDeclaration%28%29
Of course, the types can then be requested from the Protocol itself.
http://avro.apache.org/docs/current/api/java/org/apache/avro/Protocol.html#getTypes%28%29
And then it is only a matter of serializing them. And actually the API
provides even a 'pretty' option.
http://avro.apache.org/docs/current/api/java/org/apache/avro/Schema.html#toString%28boolean%29
I will definitely look at it (ie with javac). Contributing it as a tool
would be nice. But I won't go into the maven land.
Regards
Bertrand
Post by Doug Cutting
There's not a tool that does this currently. Note however that
existing tools will generate Java classes for each type in an IDL
file, so if you're only using Java then you might not need a .avsc
file for each type in the IDL.
It would not be hard to add a tool (or an option to an existing tool)
that wrote a .avsc file for each type in an .avdl file. One could
also add Maven support for this. If this is of interest, please file
an issue in Jira.
https://issues.apache.org/jira/browse/AVRO
If you're willing and able, please provide an implementation.
Otherwise hopefully someone else will help out.
Cheers,
Doug
Post by Jeremy Kahn
The "types" field in a protocol (.avro) may get you what you need. The
corresponding schema objects should be able to render to well-formed
avsc
Post by Jeremy Kahn
objects.
Post by Bertrand Dechoux
Hi,
I have lots of avro schemas and most of them are about complex objects
(ie
Post by Jeremy Kahn
Post by Bertrand Dechoux
nested definition of stuff). The syntax of avro idl is attractive in
order
Post by Jeremy Kahn
Post by Bertrand Dechoux
to build something more readable and thus maintainable. However, it
looks
Post by Jeremy Kahn
Post by Bertrand Dechoux
like I can't generate any avsc from a avdl (or the avpr generated from
the
Post by Jeremy Kahn
Post by Bertrand Dechoux
avdl). I understand what is a protocol and I don't need one but the idl
syntax is really attractive. Is there really no way to use it for that
purpose?
Regards
Bertrand
PS : I remember seeing a discussion about that subject but I can't find it.
--
Bertrand Dechoux
--
Bertrand Dechoux
Bertrand Dechoux
2013-06-18 22:05:20 UTC
Permalink
Thanks for the review. I updated the patch accordingly.

Regards

Bertrand
Post by Bertrand Dechoux
Waiting for a review : https://issues.apache.org/jira/browse/AVRO-1337
Regards
Bertrand
Post by Bertrand Dechoux
After browsing the (nice) API, it seems indeed trivial.
The Idl class allows to read/parse the related file.
http://avro.apache.org/docs/current/api/java/org/apache/avro/compiler/idl/Idl.html#Idl%28java.io.File%29
The Protocol object can then be requested from it.
http://avro.apache.org/docs/current/api/java/org/apache/avro/compiler/idl/Idl.html#ProtocolDeclaration%28%29
Of course, the types can then be requested from the Protocol itself.
http://avro.apache.org/docs/current/api/java/org/apache/avro/Protocol.html#getTypes%28%29
And then it is only a matter of serializing them. And actually the API
provides even a 'pretty' option.
http://avro.apache.org/docs/current/api/java/org/apache/avro/Schema.html#toString%28boolean%29
I will definitely look at it (ie with javac). Contributing it as a tool
would be nice. But I won't go into the maven land.
Regards
Bertrand
Post by Doug Cutting
There's not a tool that does this currently. Note however that
existing tools will generate Java classes for each type in an IDL
file, so if you're only using Java then you might not need a .avsc
file for each type in the IDL.
It would not be hard to add a tool (or an option to an existing tool)
that wrote a .avsc file for each type in an .avdl file. One could
also add Maven support for this. If this is of interest, please file
an issue in Jira.
https://issues.apache.org/jira/browse/AVRO
If you're willing and able, please provide an implementation.
Otherwise hopefully someone else will help out.
Cheers,
Doug
Post by Jeremy Kahn
The "types" field in a protocol (.avro) may get you what you need. The
corresponding schema objects should be able to render to well-formed
avsc
Post by Jeremy Kahn
objects.
Post by Bertrand Dechoux
Hi,
I have lots of avro schemas and most of them are about complex
objects (ie
Post by Jeremy Kahn
Post by Bertrand Dechoux
nested definition of stuff). The syntax of avro idl is attractive in
order
Post by Jeremy Kahn
Post by Bertrand Dechoux
to build something more readable and thus maintainable. However, it
looks
Post by Jeremy Kahn
Post by Bertrand Dechoux
like I can't generate any avsc from a avdl (or the avpr generated
from the
Post by Jeremy Kahn
Post by Bertrand Dechoux
avdl). I understand what is a protocol and I don't need one but the
idl
Post by Jeremy Kahn
Post by Bertrand Dechoux
syntax is really attractive. Is there really no way to use it for that
purpose?
Regards
Bertrand
PS : I remember seeing a discussion about that subject but I can't
find
Post by Jeremy Kahn
Post by Bertrand Dechoux
it.
--
Bertrand Dechoux
--
Bertrand Dechoux
--
Bertrand Dechoux
Loading...