Data-Driven documentation + consumption in tools?

Pipelines
1

Hey fellow Techies and hope you are having great time at E3, if you went there!

Intro

I’m probably gonna start documenting most of the tools I’ve written soon.
I really like Data-Driven programming, I use it whenever it’s appliable.

For those who do not know data driven programming; say that I have a sortable, filterable asset browser. I want to consume same code for photoshop and maya. The flow of the application changes depending on different data, where data is either xml / ini / json or from db etc.
The data could say only show .psd files for photoshop and .ma / .mb files for maya or only search in x, y folders for the assets etc.

Documentation

I want to apply the same rules to documentation, and potentially consume some parts of the documentation in the UI. I dont’ want to document stuff just because noone will ever read it.

I want to consume parts of it in tooltips, in help buttons, help centers etc, related topics etc.

So data driven documentation is what I had in mind.
So I will have an XML file containing the documentation

Example Documentation Data:


<documentations>
	<documentation xml:id="PhysX">
		<title>NVIDIA PhysX dynamics</title>
		<author>Yasin Uludag</author>
		<tooltip>this is the tooltip that will be shown</tooltip>
		<description>we should have a long description here</description>
		<howto>
			<step>1. First</step>
			<step>2. Seconds</step>
			<step>3. Last</step>
		</howto>
	</documentation>
</documentations>

Now the idea is to be able to consume this DATA at the same time as using it for DOCUMENTATION.

Say that the documentation is HTML code.
To satisfy that: documentation could be generated using XQuery expressions.
For those who do not know: XQuery is to XML what SQL is to database tables.

Qt has a module called QtXmlPatterns, which allows us to use XPath, XQuery, XSLT and XML Schema validation

I hope you are with me up untill now!

Example XQuery expression:


declare variable $inputDocument external;

doc($inputDocument)/documentations/documentation/
<generated>
<p><span style=" font-size:11pt; font-weight:600;">
Title: { string(title) }
</span></p>
<p> Author: { string(author) } </p>
<p><strong> Short description: </strong> { string(tooltip) } </p>
</generated>

which unfolds to, after using QtXmlPatters module:


<generated>
<p><span style=" font-size:11pt; font-weight:600;">
Title: NVIDIA PhysX dynamics
</span></p>
<p> Author: Yasin Uludag</p>
<p><strong>Short description: </strong> this is the tooltip that will be shown</p>
</generated>

So now I have generated the HTML code required for the documentation content, where it can contain long descriptions, detailed steps, links, embedded movies etc.
This output can be injected to any html documentation code. I didn’t generate all of the elements from our original XML, wanted to keep it simple!

Also the original XML only contained one entry of documentation, XQuery expressions runs for every entry it finds with the matched pattern so the expression will never grow once it has been estabilished!

The plan now is to CONSUME the original XML file in our UI by making an interface class that allows us to query and adapt the data! Smallest example would be tooltips.

So summary:

Xml contains all documentation.
XQuery expressions generates HTML code for documentations with all different properties using a pattern. Could also generate entire HTML pages and upload to Wikis.
Documentation interface / adaptor class allows us to query tool tips, short descriptions and other useful doc to our UI from the XML and not the wiki / html

Feedback:

I’m gonna be open, it will be my first time documenting code, I’m very open to feedback. I don’t want to document something that goes to waste because noone cares. I want to document and be able to make it apart of my UI! So I sat down and thought how I could do this, and this method that I explained blossomed.
Still, feedback is very important to me :slight_smile:

Best regards Yasin!

2

I’ve blogged about documentation here: Documentation is a tech possibility, not a project management hurdle | RobG3d

Overall, your concept is right (current documentation practices leave lots to be desired), but I think your current implementation is missing a robust ‘why.’ There are a few questions ‘why’ here that would be useful to answer.

Why are current documentation practices lacking? Because they do not integrate the documentation into the program. So we need a system to integrate documentation into programs.

Why else? Because documentation is a bitch to keep up to date.

Why is it a bitch to update? Because the source of it is often disconnected, and no one reads it anyway.

Why should developers use your solution to these problems? Because it is a low-impact solution that integrates documentation with minimal work required.

How many of these 'why’s does your implementation answer? Here are the problems I have with it:

  1. It is limiting to documentation set up in this specific way.
  2. The documentation is too difficult to edit realistically without an additional tool, which means you’re going to run into the same problem of no one keeping docs up to date.
  3. It actually has nothing to do with techniques for integrating the documentation into the tools- you have two very different concerns here: the presentation/integration of documentation into the UI, and the collation of documentation data for consumption. The two should be entirely abstracted.

So what I’d do is, solve one problem first- ideally, the presentation/integration problem- and abstract it away as much as possible from the data collation solution (you can use a text file- it doesn’t matter). Then use those integration needs to figure out your data collation solution- which, I think you’ll find, can be much better than the (IMO) overly-engineery, and actually pretty mundane, XPath solution (I know you can do better!).

Actually if you have any interest in making this open-source and in python, I’d be willing to try to help (and provide some oversight like when we solved basically the same data-driven, tightly-integrated documentation solution at BioWare). I want to implement a similar thing at CCP so it could be a useful start.

3

Hey Rob!

Confirming that I’m on the right approach is nice to hear :slight_smile:

I read the blog post, was very insightful! Thanks alot!
This is why I love feedback, production experience feedback, something I am low on, need more!

I agree with all the “why” points you made, xml will be hard to edit, besides noone will ever do that. It’s also not futureproof since if I ever leave the company i’m having my intern at, noone would know how to document / generate, unless I teach them (bad idea).

We would need an editor for this as you mention to make it more productive, but a wiki just like you described in your blogpost already acts like one, and everyone can commit to it, it’s easy to edit and format! Or should I say, one possible editor, since data should be retrievable from any source :wink:

I guess I was looking at the problem(s) from a wrong perspective. The entire design just got alot more simple than what I initially had =}

I’m defo up to make this an open source project, and would love and appreciate any feedback during the development!

Thanks again Rob, we can talk more about the open source thing in private if you want :slight_smile:

PS: Going to make good use of “why’s” in the future to pin down and abstract problems…

4

I’ll keep the design public, the more ideas the betters aye? :slight_smile:

768×283 47.4 KB

This is what I have so far, have no implementation yet, only spawning on ideas.
I wanted to treat the entire thing as a content pipeline, where the wiki is an asset, then you have auto exporters (the grabbers and formatters) which prepares the data.

The grabber formatter suits could be registered using external data sources and loaded at runtime, or could just have a manager singleton that has them registered.

Then we have the importers (Documentation model and sort filter model) which queries data depending on keywords, inputs etc. Allows same windows to consume the same documentation differently using filter and sort, where one could be descending and the other ascending etc :slight_smile:

Another idea I had was, what about making the service layer a server that allows TCP / IP communication by sending messages such as give me documentation about this keyword from this source; the server could then send back the information wanted, allows any tools regardless of language to consume the data, which is ideal imo, and something I want to strive after! Because I develop tools both in C++, C# and Python.

Though I’m abit concerned about the documentation model ( the interface towards the data ).
What kind of hints did you give your widgets so they could go and fetch the data?

Did you give hardcoded links to specific wiki pages for your widgets (don’t like this) in the visual designer?

Did you use some custom properties on your widgets that was used as hints on the querying of data (I like). But with this approach, how can you be sure that the retrieved data is correct and valid?

How did you handle dynamically generated widgets? When doing data driven programming, I have some UI that is generated depending on the data so I can consume the tool differently, how would you approach something that is dynamically generated? I have ideas that is simple, but I want to hear out other ideas aswell :slight_smile:

Why did you choose visual designer as a dependency?

I have tools both in C#, C++ and Python and want to consume the documentation in all of them. Hence the idea about making the server approach with TCP / IP communication, true decoupling… :slight_smile:
This decoupling with TCP / IP offers advantages such as visual designer and qt designer can now be used to mark hints on widgets, and if no hint is found, it uses label / name to consume the server services in the language of your choice!

Would love to hear more about this from anyone!

Best regards!

5

[QUOTE=LoneWolf;10903]I wanted to treat the entire thing as a content pipeline, where the wiki is an asset, then you have auto exporters (the grabbers and formatters) which prepares the data.[/quote]
Yes, this is good. We didn’t have this service middle layer, and it caused issues when we had 250+ simultaneous connections to the Wiki- it actually brought the whole thing down and caused a massive amount of headache. Having a single service, with crawlers that updated the content, would be safer and faster (since the transformation of the docs wouldn’t have to happen on the client, it could happen on the server and just be sent straight to the client).

The grabber formatter suits could be registered using external data sources and loaded at runtime, or could just have a manager singleton that has them registered.

Lots of ways to do a plugin system, as long as it is truly a plugin system (does not require code registration of alternative implementations). You’ll also want a way to register multiple sources for the same implementation. You could have this auto-updating run live (hard to do), or what would probably be easier is a local process that could communicate to the service to tell it to look for new plugins and new sources for plugins using some IPC (probably wouldn’t want this as part of the service API).

Then we have the importers (Documentation model and sort filter model) which queries data depending on keywords, inputs etc. Allows same windows to consume the same documentation differently using filter and sort, where one could be descending and the other ascending etc :slight_smile:

This is great. It will be more clear once you start to figure out your service API which you can do once you start to figure out your client needs. I think this sort of project is best driven top-down (from the needs of the client), rather than bottom up (the design of the data).

Another idea I had was, what about making the service layer a server that allows TCP / IP communication by sending messages such as give me documentation about this keyword from this source; the server could then send back the information wanted, allows any tools regardless of language to consume the data, which is ideal imo, and something I want to strive after! Because I develop tools both in C++, C# and Python.

Again, great. Formalize a documentation format and use a REST API to communicate to clients.

Though I’m abit concerned about the documentation model ( the interface towards the data ).

You should design this once you have clearer needs- like I said earlier, I think this should be done top-down. Consider model after view, since the view is going to drive what data we need and how. This is the difference between exposing an internal and external API. A good external API needs to walk the line between providing a clean exposure of data/functionality directly (like an internal API), while also providing a task-oriented API (give me all data that matches this criteria). You don’t want to require clients to have to replicate the server environment to do what they need.

I expect when you start considering client needs, you’ll figure out something like: 1) We want to use html as a common way to represent content display, so all the source data should transform into html if not already (and the html should be normalized according to some service specs), 2) We’ll need some way to grab a ‘summary’ from the content, 3) We’ll need to support links to specific areas of content, not just a block/page of content, 4) Some form of keyword support… questions like these that may be easier to think about when coming from the client’s perspective.

What kind of hints did you give your widgets so they could go and fetch the data?

We gave them a uri. Plugins were set up to handle a uri scheme (http://, wiki://, doc://, etc.). So you couldn’t have conflicting schemes. We could have gone more specific (confl:// for Confluence instead of wiki://), but these are things that you can change when you’re developing an inhouse app.

Did you give hardcoded links to specific wiki pages for your widgets (don’t like this) in the visual designer?

The ‘link’ was a uri, so in effect it was hard-coded (though not to an http page or local file, it was hardcoded to a uri that the plugin could transform to find the actual resource). This could easily be abstracted for a data driven approach where the hard-coding is to a target in an xml/config file that defines the actual link. Or to a DB table/row instead of xml, it doesn’t matter.

Did you use some custom properties on your widgets that was used as hints on the querying of data (I like). But with this approach, how can you be sure that the retrieved data is correct and valid?

Not sure what you mean.

How did you handle dynamically generated widgets? When doing data driven programming, I have some UI that is generated depending on the data so I can consume the tool differently, how would you approach something that is dynamically generated? I have ideas that is simple, but I want to hear out other ideas aswell :slight_smile:

The basic implementation was, you can add a ‘tooltip’ to your UI, and you register controls with the tooltip (same as a normal tooltip), that registers it with the collabdoc system. The ‘register’ method takes the control and the uri. It works fine with more dynamically generated UIs (though I guess it depends on your definition of ‘dynamic’).

Why did you choose visual designer as a dependency?

It isn’t a dependency, it is just a plus. You can add a collabdoc hookup for a control through the Designer, just as you can add a tooltip for a control through the Designer.

I have tools both in C#, C++ and Python and want to consume the documentation in all of them. Hence the idea about making the server approach with TCP / IP communication, true decoupling… :slight_smile:
This decoupling with TCP / IP offers advantages such as visual designer and qt designer can now be used to mark hints on widgets, and if no hint is found, it uses label / name to consume the server services in the language of your choice!

Just remember that the service/backend portion is probably going to be the easy part. The first thing to do, IMO, is figure out how the client portion is going to work (how it is going to integrate and work in the UI). And remember that all of the client code is going to have to be repeated in every language, in every target platform. It is by far going to be the most difficult and expensive part of the implementation process- and if you’re anything like me, the least exciting!

6

Lovely!

[QUOTE=Rob Galanakis;10904]
Just remember that the service/backend portion is probably going to be the easy part. The first thing to do, IMO, is figure out how the client portion is going to work (how it is going to integrate and work in the UI). And remember that all of the client code is going to have to be repeated in every language, in every target platform. It is by far going to be the most difficult and expensive part of the implementation process- and if you’re anything like me, the least exciting![/QUOTE]

I agree very much about this, the doc fetching and preparing is very easy to handle, infact the UI part is also easy. But when you want to expose it with a clean API to the user, in several langauges and platforms, that’s where the problem starts, unless we love implementing several times :smiley:

I’m fine with that, but what if I change the implementation? Now I have to change it on all platforms. Seriously an issue.

So i’m with you about that it’s gonna be the most boring / hardest part. Besides, if we want to add a new feature, say an animated tooltip with embedded browser code, we will have to do it in WPF, Winforms, Qt etc. It’s gonna be hell to update each platform.

We agree that we need atleast some code in C++, C#, python, LUA to integrate the API for fetching prepared documentation from the server… but we also agree on that we want to keep the UI integration at minimum and not deal with doing the same implementation several times.

So the API has to be written for each platform, but we seriously don’t want to deal with the UI integration aswell. Just barebones to show something when requested.

So what we really want and beg for is, decouple the UI integration from the users application, and use other means of communication to show that complex fancy tooltip with links and references, communication such as TCP / IP or PIPES.

SO… I’m thinking, maybe have another layer of service, LOCAL, like a process. a UI service decoupled from the users application which has Qt or any other UI API embedded inside it.

This thing is running in the background and communicates through IO PIPES to the USERS applications through the exposed API to them.
Whenever this service is offline, the tooltips don’t show up. So this service will recieve messages through the API like pop tooltip at x, y with documentation z, anchor point at u, v and width, height is h, w, and then it will spawn a Qt widget!
Or even a Qml window (Qt version of Xaml for WPF, highly customizable and very animateable, pure beauty!)

So instead of having to implement the complex tooltipping and other complex UI documentation views in each platform, we implement it once and pop them by messaging in PIPES with “always on top” properties etc.

So we have two services now, the documentation service and the ui service.
If you think about the Model-View-Controller pattern, our documentation service would be the model, our ui service would be the view, our controller would be the API we EXPOSE to the user which takes inputs and does something :slight_smile:

So the API would only need to have some sort of register method, that keeps track of each widget for each platform and then handles the rest of the UI integration with this local process by messaging to it. Atleast the complex tooltip part. The user can still go with using the regular tooltipping system in each platform that only contains pure text, by requesting only text documentation. This UI Service will only be responsible about showing compelx tooltipping systems, so I dont’ have to implement it in every language I want to support!

So we have:
UI Service (view)
Documentation Service (model)
API (controller)

Those are all decoupled from each other, they are also decoupled from the users application except the API he uses. The user only sees this API he has, then the API handles the communication by TCP / IP for Documentation, and PIPES for the local UI Service. Decoupling is the most important part for me, so I don’t have to implement it 4, 5 times when I add, change or remove a feature, or when I fix a bug.

So basically we have the MVC pattern outside the application where the API takes inputs and commands, fetches something from the model and does something on the view.

This needs some more thought before an attempt at implementing.

What do you think of this? Any unforseen circumstances that can arive?

7

Yes, you’d just want to return a json object or something from the server API- this json can then be wrapped by a pretty simple module written in any language. Think how the twitter, google docs, wordpress, etc APIs work- each language implementation is pretty similar, just wraps the data from the service into a class/object, and exposes some methods that under the hood call the service methods. There’s no real way around this.

Your idea for the UI service is interesting, though. If it worked, you could feasibly not need the docserver/client language-specific API’s, you’d just need the uiserver/client language specific APIs.

I have no idea if it’d work or not, I haven’t done anything like it- I’ve just used pipes for blocking calls (like lua opens a pipe to python, python displays a messagebox via python.net), and a basic service. I imagine it will be incredibly complex to do well, but it is worth a try. It is a completely separate project, though, a way to bind together UI’s in different languages. It may be significantly simpler to just re-implement it for the most common languages. You need to decide what the focus is- the cross-platform UI, or the data-driven documentation.

8

What I had in mind was kinda something like that =}

About UI: I agree it’s defo a separate project :slight_smile:

I think the most solid would be to start on the data-driven documentation project and then release it together with a pyqt example of how we would integrate and consume it in a custom tooltip! People can make their own implementation for their platforms since I highly doubt I’ll have time doing it for everything out there. Only time will tell, alot can change during development, maybe it’s easier than I thought :slight_smile:

Then maybe i’ll give this ui service that runs locally a chance, make some extentions to the original API and communicate to the ui service to spawn Qt / Qml stuff on the fly at the requested locations and then fade them away / kill etc!

Gonna throw up an online repository for now!

9

Then maybe i’ll give this ui service that runs locally a chance, make some extentions to the original API and communicate to the ui service to spawn Qt / Qml stuff on the fly at the requested locations and then fade them away / kill etc!

Actually that’s a pretty awesome idea and would be simple enough, probably for the scope of the project. But either way, like you said, we’d need a Qt ui implementation.

BTW can you use Mercurial for the repository? Host it wherever you like.

10

yeah sure =}
I’ll pm ya when it’s up!

11

Ughuhuu, project name V_V

I hate this part, will take me forever! Any ideas?

12

Alright been playing around with the Pipe idea I had for the tooltip integration cross language / cross platform and it works =}

768×283 22.2 KB

If you have been browsing around a homepage and been noticing that some keywords are hyperlinks, and then when you hover over them, a tooltip pops up as an advertisement, what I’ve done is pretty much the same and it will work cross langauge because the Qt part is a completely separate process, and you will only show, hide the tooltip by sending messages to it! This can also be TCP/IP based on localhost, as long as it’s local this works!

So I have a local process running with a Qt application embedded inside it, which has a tooltip widget created but hidden.

Then we have our real application, communicating to this ui service application through a pipe.

So to consume the ui service tooltip, we just have to use a single method right now. This sends a message through the pipe into the service application. The message is either:
SHOW, x, y, w, h
or
HIDE, x, y, w, h

which shows or hides the tooltip at the desired location. The tooltip is made in QML (a declarative view, which uses basically a text file to define the look and databinding), again QML is what XAML is for WPF.

So people will be able to customize the look of the tooltip 100% to what they desire by some QML scripting. Animating in QML is very easy to do and fun, which means instead of spawning it like in an instant, we can fade it in, scale and animate etc.

Though this means that, if the service is not running, the tooltips stop showing and popping :slight_smile:

Here is a video as proof of concept for the spawning :slight_smile:

About the documentation service layer progress, I haven’t started on that. Want to talk with Rob before making any decisions, as I have a few more questions that I would like to have cleared up for me!

Anyway now that the UI integration problem is solved, the rest will be smooth to implement so it can be consumed by any tool in any language by anyone. =}

Cya around!

PS: The image doesn’t show the design or anything related to the final product, it’s just there to explain how this works… who knows how the structure of the services will look once the " documentation service " starts its development phase!

13

I had time today to work on the wiki grabber…

I can grab pages / sections of a wiki as html files now (intermediate documentation)

I can also grab the entire wiki page as an image with QtWebKit, don’t know what the uses for this would be though…

Using the MediaWiki api through python to query and parse sections and pages, allowing me to create the intermediate documentations…

This video demonstrates temporary grabbing class, which will later be running on it’s own server exposing RPC or RESTful principles… Right now it’s only local:
http: http://www.youtube.com/watch?v=X69lhhm6P0A

I’m still experimenting with how I want to expose the methods to the client. I’m thinking about going dual mode, RESTful and RPC. RESTful for the client (readonly using GET) and RPC for low level interaction that requires authentication, such as controlling server behaviour etc…
see: http://blog.killtheradio.net/technology/rest-vs-rpc-and-why-authentication-is-the-deciding-factor/

I’m gonna give this some thought and hopefully have something to show soon.
I hope to show soon a simple application that uses the UI and Doc services to grab documentation from a wiki and show a complex interactive tooltip usign either keywords / widget names / labels etc… :slight_smile:

And the UI service again is cross platform / language, it’s also independent of the documentation service… So I will pretty much make a small API that allows you to use these services immiidately when you run them on the computer!

Again, still giving thought on the RPC / RESTful services so there is nothing of that implemented yet, I will start on the services as soon as I have nailed the grabbing, exposing and consuming part locally! :slight_smile:

For those who missed how the tooltip will work, here is a video of it spawning above a button with garbage inside:

So instead of garbage, it should contain interactive elements with the documentation grabbed from a wiki source!

14

Hey, been a while! Didn’t have that much time to work on this, my intern was comming to an end and deadlines where approaching for the company.

Though the good news is I got employed as a Tools Engineer now, and I was supposed to have an intern as a TA :P, just couldn’t get my fingers away from the engine asset compiler, exporters, tools… Now I’m welded into the pipeline holding everything together :slight_smile:

Anyhow enough off-topicness, just wanted to share that I got a job now =}

Here is the progress I made today about this tooltip service. You can register widgets with keywords, if no keyword is used it uses the text property of a widget… still alot to do since I didn’t have time to work on this for about an entire month :slight_smile:

Video: http://www.youtube.com/watch?v=32CzvYgNmYc

The content shown in the tooltip services QML window has already been parsed and saved locally by the MediaWikiGrabber class in a separate process :slight_smile:

Enjoy :slight_smile:

15

Congrats on the new job! I’ve followed some of your work on this site, and as I’ve done so, I’ve been continually impressed by your thorough grasp of the subject matter, prodigious work ethic and overall attitude. I’ll continue to expect great things from you. :):