How to… find out which templates are being used by which pages

There’s no clear way in MindTouch to figure out which templates depend on other templates, nor which templates are being used at all. MindTouch offers a Lucene-based search feature, but as far as I can tell, DekiScript code (including calls to templates) are excluded from the indexing, so they aren’t searchable.

Fortunately, all MindTouch pages are stored in a fairly simple table in the MySQL database. If you have access to the underlying database, you can quickly assemble the necessary information. When a page “uses” (or calls) a template, it must call it by the page’s title (minus the “Template:” part). There are a number of ways to do this:

• {{ template(“template_name”); }}
• {{ template_name(); }}
• and others

Fortunately, in all cases, the part after “Template:” must be in there. All this information is stored in the “pages” table. The title (without the “Template:” part) is in pages.page_title, and the actual page content, with the UNINTERPRETED DekiScript is in pages.page_text. Putting all this together, I wrote the following simple SQL statement:

SELECT t.page_title, p.page_title 
  FROM pages t, pages p
 WHERE t.page_namespace = 10
   AND NOT t.page_title LIKE ''
   AND LOWER(p.page_text) LIKE CONCAT('%', LOWER(t.page_title), '%') 
 ORDER BY t.page_title;

There are two other details going on here that deserve mention. The first is that the query is doing a cross product of the pages table with itself, where the “t” pages have the “page_namespace” of 10. I’m not sure if this is a MindTouch constant, but the namespaces are those special page name prefixes you encounter in URLs: “Template:”, “Special:”, “User:” and the default (nothing). In this case, “page_namespace = 10″ is picking out all the “Template:” pages for the “t” (or template) pages. Also, DekiScript is not case-sensitive. So in order to grab all matches, I cast everything to lowercase for the comparison.

This query doesn’t list the pages that have NO matches (i.e. the ones you can very safely delete), but could be modified to do so.

This is NOT a foolproof solution. Any page that even mentions a template, or happens to have a name that’s a substring of another template name might return false positive matches. The query could be improved with further effort, but for me at least, it’s easier just to go through the results by hand. What’s important is that we aren’t missing pages that are using a template.

Helping Hand: Ignoring Closed issues in TicketForm

Chamila is playing with DekiScript, and trying to use an example of the TicketForm table (for our issue tracking) that doesn’t show issues that have the status “Closed”. Currently, the page is giving the following error:

/content/body/div[3]/pre, line 299, column 20: invalid Primary (click for details)

This means, of course, that there’s something wrong with the DekiScript. The “invalid Primary” means it was looking for a number where there’s some other data type. Could be a syntax error, or it could be that a variable or literal value of the wrong type is being used where it shouldn’t. Chamila has helpfully already pointed out to me the line that he was working with:

if(param is not ["Closed"]) { "xxxxxxx" } else { "YYYYY" }

There are a couple of syntactical errors here, so these should be tackled first. One problem with this statement is with the logical test in the “if” clause:

param is not ["Closed"]

If you look in the list of DekiScript Operators, you will see that there is no “is not” operator. There actually is an “is” operator, but it tests for a match on the data type of two values, rather than for equality of their values. Furthermore, there is no “not” operator. Instead, you must indicate negation using either the “!” operator in combination with the comparator “==”, or the combined operation, “!=”:

param != ["Closed"]

Next, we have to look at what is being compared here. On the left is the param variable (more on that below). On the right is a literal, and a valid one at that. But what kind? The brackets ([ ]) indicate that the value being compared is a literal List. Contained within that list is a single value, a str (string) with the value “Closed”. My guess is that the test is trying to find parameter values that match “Closed” (the status we are looking to exclude), so in this case, the list is incorrect:

if(param != "Closed") { "xxxxxxx" } else { "YYYYY" }

Now, when I save the page, the syntax errors are resolved. Instead, we see the following error:

/content/body/div[3]/pre, reference to undefined name ‘param’: line 299, column 20 (click for details)

The problem now is just what it says: the param variable that we’re comparing hasn’t yet been defined for our scope. To see why, you have to look at the code around it:

<tr id=(p.properties.id.text) class=(class)>
if(param != "Closed") { "xxxxxxx" } else { "YYYYY" }
foreach(var param in params){
  if(param.value.show && param.value.show == 'ticket')        continue;    // if not supposed to show in table, skip.

Notice that the param variable isn’t defined until the foreach loop in the following line. This is what’s causing the new error. When I switch the lines, the error disappears:

<tr id=(p.properties.id.text) class=(class)>
foreach(var param in params){
  if(param != "Closed") { "xxxxxxx" } else { "YYYYY" }
  if(param.value.show && param.value.show == 'ticket')        continue;    // if not supposed to show in table, skip.

Now we see the table, as follows:

The table only shows X's (there are no matches)

What’s happening? There are only x’s across the top (there were some in the table cells, too, but I removed an unrelated line of code for the purposes of this tutorial) (note also that they appear at the top of the table because we are writing the text out between a <tr> tag and a <td> tag, rather than within the <td>). As you can see from the if statement, this means that in no case is param “Closed”. Even when we have a Status column with a value of “Closed”, it’s not matching. This is because the if statement is trying to compare the param value (which is some sort of object) to the str value “Closed”. The first hint on how we might want to do things is in the line following the if statement:

if(param.value.show && param.value.show == 'ticket')

The param variable has a “value” attribute, which in turn has a “show” attribute, which can have a value of “ticket”. The whole structure and meaning of the param variables is specific to the TicketForm templates, but the point is that this is an object, and not just a simple string. Rather than try to guess where the value is that we’re testing, I’ll skip to the important part. The param here marks a type of parameter (i.e. a column in the table: Type, Urgency, Status, and so on). This is generic, and doesn’t contain the value we need, just the name of the parameter that contains the value.  Looking at the next line down, we see:

var property = Map.values(map.select(p.properties, "$.key=='"..String.replace(String.toLower(param.key)," ","_").."'"))[0];

The code is grabbing a page property value with a name equal to the param.key value, and assigning it to the property variable.  Further down, we see:

if(param.key =='title'){
  <a href=(p.uri) style=(param.value.style.table)>property.text </a>
}else{
 <span style=(param.value.style.table)>property.text</span>
}

So, it looks like we could get the value from property.text. I move out if statement to after the property declaration, and replace param with property.text:

foreach(var param in params){
  if(param.value.show && param.value.show == 'ticket')        continue;    // if not supposed to show in table, skip.
  var property = Map.values(map.select(p.properties, "$.key=='"..String.replace(String.toLower(param.key)," ","_").."'"))[0];
  <td>
  if(property.text != "Closed") { "xxxxxxx" } else { "YYYYY" }

I hit save on the editor, and get the following:

The Closed Status value has Y's

You can see that all the cell values now have a set of x’s before them, all except the “Status” cell with the value “Closed” (note that I also moved our if statement to within the <td> declaration so that we could see the result next to the values being tested). So, now we’re most of the way there! I leave it up to you guys to do the rest:

1. Only test against the parameter value if the parameter type is “Status”
2. If the row has a “Closed” status, skip the row entirely (rather than printing out test values)
3. Bonus credit: print out a count of the number of issues, but don’t count tickets with a “Closed” status

How to… convert objects to Strings and vice-versa

In most languages, you would convert an object to a String by using something like a “toString()” method on the object itself, or calling a “String.valueOf(obj)” method. There’s also implicit conversion, which in a lot of languages works like this:

String mystr = myobj + ""

In DekiScript, you can do both implicit and explicit conversion of objects and other non-String values (e.g. numbers and dates) to Strings. However, neither works the way one would expect if you’ve worked in other languages that work like I’ve described above.

Explicit Conversion

In DekiScript, this is done via a method on the String utility class. The method is correctly, but somewhat counter-intuitively called String.serialize(obj). The reason for this is that when you call this, rather than some human-readable text for screen display, you get a complete description of the object. For example, if you call String.Serialize on a page.properties object:

<p>"Full page.properties object: "..String.Serialize(page.properties)</p>

you might see the following:

Full page.properties object: { team : { date : "Thu, 04 Feb 2010 23:30:36 GMT", editsummary : "", language : nil, mime : "text/plain", name : "team", namespace : "urn:custom.mindtouch.com", revision : 2, revisions : "local://a5277d3e7d07422ece3d3759090b18f8/deki/$proprevisions", size : 41, text : "[{\"members\":[\"bryan.menell\",\"tim.high\"]}]", xml : nil } }

Although it might be ugly, the nice thing about this is that you see all the data, and more importantly, you can turn that String back into the object at a later time using the String.Deserialize() method. Note also that in most cases, an object will be nothing more than a composition of Maps and Lists, so you should be able to get at some human-readable text if you manage to dereference them correctly.

Implicit Conversion

Automatic conversion to Strings can happen under certain circumstances, but I haven't yet researched enough to find out when and how. One thing I can tell you is that when you place an object or variable between HTML tags, there is no need to call String.Serialize. For example, with:

<p>page.properties</p>

you get:

{ team : { date : "Thu, 04 Feb 2010 23:30:36 GMT", editsummary : "", language : nil, mime : "text/plain", name : "team", namespace : "urn:custom.mindtouch.com", revision : 2, revisions : "local://a5277d3e7d07422ece3d3759090b18f8/deki/$proprevisions", size : 41, text : "[{\"members\":[\"bryan.menell\",\"tim.high\"]}]", xml : nil } }

However, as you can see in the first example, use of concatenation forces you to have to perform explicit conversion (in testing, I got a blank value for the properties, rather than seeing an error - very confusing).

How to… update a page property using DekiScript

Editor Note:

The information presented below is INCORRECT. Since this post, I have learned the following things:

1. Yes, you can update page properties in the way described below, but it is NOT a permanent change – you are changing a copy of the persistent values
2. In fact, the scope of this change is localized to your current block. In fact, I tried making a template to do this, and the change doesn’t survive the template call.
3. The ONLY way to change page properties (and tags and other values) is via an Ajax call (i.e. via REST)

I actually spoke over the phone with Steve Bjorg, the CTO of MindTouch, about this subject. He gave me an answer, but I’m not quite sure I buy it. The idea is that they wanted to ensure that page loads were “idempotent” (that is, every time you load a page, you get the same thing). I’m pretty sure there are times when you might not want that to be true, and in fact, there’s a very easy workaround to this: just update the properties using Ajax. That’s what I had to do when I started using certain properties and tags as search “handles”. At any rate, our conversation on this subject was very brief and he admitted that he wasn’t remembering exactly why off the top of his head. I’m looking forward to a more complete explanation (or, even better, a change to the DekiScript API!) some time in the future.

So, the code below “works”, and is informative about how Maps and the page object works. Just keep in mind that it’s not very useful…

——

This should be easier to do than it is. Don’t ask me why.

There are two things that make updating a page property more complicated than it has to be:

1. Maps (and Lists) can’t be updated by simple assignment (in the case of DekiScript, the “let x = y” clause)
2. Not only are page properties a map… the page itself is a map!

When you update a map in DekiScript, instead of just assigning a new value to the map specified, you have to “append” a new key/value pair to the map (using the pre-existing key!) via the concatenation operator (‘..’):

let myMap ..= { key-to-update:new-value };

For more information on working with Maps and Lists in DekiScript, see this reference.

Now, remember that I said a page property is a map within a map? This means that in order to update the property of a page, you need to “append” the new value to the property map, then “append” the property map itself onto the page!

var props = page.properties;
let props ..= { property-name:new-value };
let page ..= { properties:props };

Personally, if I had invented this language, I would now be sticking a fork through my right hand as punishment. So, let’s be thankful I didn’t. You can see a working example of this on the following page in our Sandbox.

How to… copy a template from another MindTouch server

It’s a little harder to create a template from the App Catalog, because there are a number of variables you have to deal with  (including in which format the source code is provided), and each app is a little different. Copying a template from an already-existing installation, however, is a lot easier. The following steps will show you how:

1. Open the template page in the source site
2. Hit the Edit button
3. Click to see the source view
4. Copy all the content from the source view
5. Cancel the edit session
6. Create a new template in the destination site (i.e. click “New page” from the http://<destination-site>/Template: page)
7. Click to see the source view
8. Select all the auto-generated content created when you added the page
9. Paste the source that you copied to the clipboard
10. Click the Source button again
11. Click Save

And that’s it! The nice thing about this is that you don’t have to worry about removing any text formatting, nor do you have to worry about changing the name of your template – this is done automatically when you copy over all the original source code.

How to… create a new Template

The following video provides an overview of how to create a new template from the App Catalog, and how and where to test it in our MindTouch installation.

How to… debug MindTouch scripts

This is a pretty big topic, but we can keep adding to it as we learn new tricks, or we can create new posts on the same topic later.

The problem with working with MindTouch is that there are SO MANY things that could make a script not work:

• Problems with DekiScript formatting
• A bug in the DekiScript itself
• An error in the Javascript
• A missing extension
• A mis-configured extension
• A missing dekiwiki tag in PHP
• A PHP error
• The dekiwiki server could be offline

This page, and the video below, will hopefully help us understand how to track down and identify the source of our problems.

Problems with DekiScript Formatting

This subject is covered in more depth in another post. Here are some signs the problem might be with the WYSIWYG formatting itself:

• The DekiScript is showing up in the page, rather than being interpreted
• You see an error like this: /content/body/pre, line 1, column 2: “}” expected (click for details)

DekiScript Errors

You should look at your DekiScript itself if you see an error on your page in red, like:

/content/body/pre, function ‘Chart’ failed (click for details)

The red error indicates it was a problem in the DekiScript interpreter

JavaScript Errors

JavaScript errors will show themselves either as a pop-up error in the page, a failed form submit, or some other dynamic page feature that just isn’t working (e.g. a button that does nothing when clicked). Most importantly, you won’t see any red error messages in the page itself, as in the errors above. Some JavaScript errors could be due to missing extensions or scripts, as discussed below. The symptoms may be the same, but the solution may be different.

There are many tools that you can use to debug JavaScript errors, but the main one is your web browser. If you’re using Firefox or Chrome, you can actually see if there are any JavaScript errors on the page by typing Ctrl+Shift+J (Internet Explorer and others may be the same way – I don’t know, since I don’t use them very frequently). On both browsers, this opens up a scripting debugger console that will show you any JavaScript errors, and where they appear on the page. Of course, your other best friends are the “View Source” option, and placing “alert” commands wherever you need to check a value in the JavaScript code.

The Chrome view of a JavaScript error

The Firefox Error Console

Note that it’s CRITICAL to understand the difference between server-side DekiScript and client-side JavaScript. Often you’ll have one generating the other, and the code can get very confusing. What you see in the browser’s “View Source” results is pure JavaScript (which may have been generated by DekiScript).

Missing Extensions

It can be hard to identify when you require an extension that wasn’t installed. Generally, it will be a message related to a library method that was supposed to be available, but isn’t. In order to identify which extension is missing, you need to look in the following places:

• If you’re writing the script from scratch, then you probably know which libraries you need. Go look in the Control Panel to see if the extension was created, and if it’s running.
• If you’re copying some templates or other code from the App Catalog, read the installation instructions for the templates carefully to see if you didn’t miss something. Sometimes you can find helpful tips in the Comments section of the application’s page.
• When all else fails, look at the library method that is giving the error, and try googling “MindTouch <library name>”, or “DekiScript <library>” or one of those combinations

Mis-configured Extension

Depending on the type of extension and the configuration error, this can show up in any number of ways. For example, if you are using the MySQL extension, any configuration error in the database (e.g. connection parameters, or even GRANT permissions on the user in the database). The best you can do in these cases is check the configuration for your extension and try to compare it to the documentation that’s available.

Missing DekiWiki Callbacks in PHP

Since we are customizing the PHP for our own MindTouch theme, there’s the potential that we do something non-standard, and perhaps leave out important functionality (this actually happened in early customization – a call to “$this->html(‘customhead’)” was removed, which didn’t cause any immediate errors, but apparently removed our ability to add Script-type extensions). If you suspect this might be the case, the best you can do is see what’s different:

• If the script works under standard themes, compare a copy of the HTML source code in both versions to see if there are important pieces of JavaScript or HTML (e.g. divs) missing from our version. Also, you can compare the actual PHP source code between themes to see what calls we aren’t using.
• If the script used to work, but doesn’t anymore, try doing a diff in Subversion to see what has changed since the last time it was working.

PHP Error

If there’s something wrong with the PHP itself, the whole web site should be broken in some way (when I tested it just now to get a screen shot, the whole site loaded as a blank page, with no error). It’s not likely that you’ll see parts of the page working, unless the error happens in an Ajax call, or the error is in a miscalculation rather that a compilation error.  When you do have a syntax error, you can check in the Apache error logs to see exactly what’s wrong. The location of these logs is different from server to server, and specifically is different on the test and public servers, but the name of the file itself is typically “error_log”.

PHP error showing up in the error_log file on the server

DekiWiki Server Offline

The dekiwiki process must be started separately from the Apache process itself. If there are any problems in the MindTouch configuration, MySQL is offline, or any number of other environment-specific problems, the dekiwiki server itself may be offline. This will be pretty easy to identify, since MindTouch helpfully informs the user (any user) that the server is unavailable (“Site settings could not be loaded”):

DekiWiki service offline

When this happens, you can try restarting the dekiwiki service (either “service dekiwiki start”, or “/etc/init.d/dekiwiki start”). If there are no lasting problems, the site should be back online pretty quickly. If not, you need to check in the dekiwiki logs to see what’s wrong. The log is located in /var/log/dekiwiki/deki-api.log. A healthy restart should look something like this:

A nice, healthy server restart

If you see any messages after the “ready” message, or instead of these lines, it is an indication of a problem in the environment configurations.

How to… insert DekiScript into a page

Demo

Words of Wisdom

Here are some quick guidelines on working with DekiScript in your WYSIWYG editor.

• Before anything, read this guide!
• Be VERY careful about formatting of text, hitting <enter> and other details
• When copying and pasting, you should ALWAYS paste into notepad or some other plain text editor to remove unwanted formatting first
• There are multiple ways to declare that text you are entering should be interpreted as DekiScript:

1. Surrounding it in {{ }} brackets, with NO FORMATTING (make sure the “Style” is set to “Normal”)
2. Surrounding it in the “DekiScript” style, with NO {{ }}
3. Surrounding it with the <pre class=”script”> tag in Source view (this is better for checking on formats than for actually working)

Option 1: Using brackets

Option 2: Using the DekiScript style

Option 3: The <pre> tag

Common Formatting Errors

A break in the DekiScript style, caused by a <p> block copy-pasted by accident

The Double Deki: Caused by a copy-paste of formatted DekiScript into a formatted block

Code meant for Source view pasted into WYSIWYG view (HTML tags become visible)

Welcome to the dev team blog

Hi team,

After working for several weeks on the MindTouch platform, it has become obvious to me that it is a VERY complex and difficult tool to work with. Although they have good tutorials (start here), in the end, most of the things you have to worry about just aren’t spelled out clearly enough. So I’ve created this blog for us to post tips to one another as we learn about the platform.

I’m finding that written communication isn’t always the best way to communicate, so whenever possible, I’ll be giving my tips via short videos. My tool of choice for this right now is Screencast-o-Matic. It’s VERY easy to learn and use, so I recommend you all give it a try and post tips here whenever you learn something new.

I’m hoping to move all this content into MindTouch whenever we get around to setting it up. Until then, this is the place!

- Tim