In the pre­ced­ing sketch, I used Math­Jax to trans­form TeX source snip­pets into nice­ly ren­dered for­mu­las. It works like this: I in­clude a TeX for­mu­la like \(\LaTeX\), and Math­Jax au­tomag­i­cal­ly turns it into \(\LaTeX\) (pro­vid­ed that Javascript is en­abled, and your brows­er isn’t old as dirt).

Now it would be re­al­ly nifty if I could mod­i­fy my Styled­TextC­trl in such a way that any oc­cur­rence of such a TeX snip­pet would be lexed and styled as TeX, not as HTML. Es­pe­cial­ly, I wish those snip­pets to stand out, to be vi­su­al­ly dis­tinct from the sur­round­ing text. Scin­til­la has, among lex­ers for many, many other lan­guages, a lexer for TeX and LaTeX and Con­TeX. Un­for­tu­nate­ly, a Styled­TextC­trl can use only one Scin­til­la lexer at a time. You have to de­cide whether your text is HTML or TeX, you can’t have both at the same time.

HTML, of course, can con­tain all kinds of other stuff: it can con­tain PHP frag­ments, or Javascript code, or CSS in­struc­tions, or for­eign XML parts con­tain­ing SVG or MathML, or CDATA sec­tions or per­haps some other stuff I have never heard of.

Scin­til­la ac­tu­al­ly tries to take care of this. Most other lan­guages Scin­til­la can lex and style are re­strict­ed to at most 32 dif­fer­ent styles (for key­words, vari­ables, op­er­a­tors, string lit­er­als, nu­mer­ic lit­er­als and so on), but the lexer for HTML has room for up to 128 dif­fer­ent style, so there is a style for Javascript string lit­er­als in sin­gle quotes and Javascript string lit­er­als in dou­ble quotes and Python key­words and VB op­er­a­tors and PHP vari­ables and so on and so on.

The right thing to do would be to look at the C++ source code of the HTML lexer of Scin­til­la and to mod­i­fy it to in­clude TeX in the list of things the HTML lexer knows about and is able to style, and then re­com­pile it and use this with­in wx­Python. Un­for­tu­nate­ly, this sounds like a lot of work, and while I’m not com­plete­ly un­will­ing to touch C++ code if there is ab­so­lute­ly no other way, I’d pre­fer to work in a more human-friend­ly lan­guage. Prefer­ably, I’d sim­ply add some Python code to my ed­i­tor to make Styled­TextC­trl be­have in the way I want to.

The­o­ret­i­cal­ly, it is pos­si­ble to im­ple­ment a Scin­til­la lexer pure­ly in Python. Un­for­tu­nate­ly, this would mean that I would have to com­plete­ly re-in­vent the HTML lexer, which sounds like even more work than sim­ply fid­dling with some pre-ex­ist­ing C++ code. And per­haps out of my league, since writ­ing such a lexer isn’t triv­ial. And prob­a­bly, it would also be slow as hell. And all I want is to style most of my text as HTML, while styling a few snip­pets as TeX.

So this is what I do: I write some code which splits my text in those parts which are HTML, and those parts which are TeX. Then I let Scin­til­la style each part in­di­vid­u­al­ly, and fi­nal­ly I glue all those parts back to­geth­er. Scin­til­la does all the hard work, and I get what I want.

Un­for­tu­nate­ly, wx­Python doesn’t ex­pose a stat­ic method to style a string: the Styled­TextC­trl just of­fers a method to style its own con­tent, but not some ar­bi­trary for­eign text. So I cre­ate two dummy Styled­TextC­trl, one with HTML as its lan­guage and one TeX as its lan­guage, and I feed a string I want to be styled to one of those two el­e­ments. Since that’s all I want from those el­e­ments, I sup­press them from being ren­dered. This is the ac­tu­al code:

You can load this code here: htm­l_with­_­tex_c­trl.py. It uses an aux­il­iary modul which sim­ply con­tains some style rules for HTML I adapt­ed to my per­son­al lik­ing, rather bor­ing stuff so I put it in a dif­fer­ent modul, you can down­load it here: htm­l_styles.py. As you can see, I also added some test code. This test code pro­duces a frame which looks like this:

It’s rather col­or­ful, but that’s be­cause I crammed a lot of test cases in a small space. A typ­i­cal source code is usu­al­ly less col­or­ful. TeX snip­pets are ren­dered with a yel­low back­ground, snip­pets with­in a com­ment are sup­pressed, non-Ascii char­ac­ters work, and the rest of the source is still styled like HTML source code, in­clud­ing all the other sec­tions with for­eign stuff. Ren­der­ing is rea­son­able fast if the source code isn’t over­ly long.

A few com­ments on the code. Well, first a few com­ments on how styling in­for­ma­tion is han­dled with­in Scin­til­la.

A text is split into to­kens, and each token can be­long to one of sev­er­al cat­e­gories of a lan­guage, like being a key­word or an iden­ti­fi­er or a string lit­er­al or a com­ment or some­thing like this. Each cat­e­go­ry gets its one num­ber, like the num­ber STC_L_DEFAULT (which is sim­ply the num­ber 0). In a sep­a­rate step, a vi­su­al style can be as­signed to one of these style num­bers, for ex­am­ple, we could spec­i­fy that STC_L_DEFAULT should be styled with a white back­ground and a black fore­ground color and a monospace font and nei­ther bold nor ital­ic.

The style in­for­ma­tion pro­duced by the lexer is stored in a string that con­tains a tan­gled ver­sion of both the orig­i­nal text and the style data. As a first ap­prox­i­ma­tion: each char­ac­ter is stored in a “cell” con­sist­ing of two bytes. The first byte is the char­ac­ter it­self, while the sec­ond byte con­tains the style in­for­ma­tion. For ex­am­ple, if the char­ac­ter be­longs to a token which is to be styled as STC_L_DEFAULT, then this sec­ond byte will be STC_L_DEFAULT, that is, the byte 0x00. For ex­am­ple, the char­ac­ter “a”, as a byte, is 0x61, so a cell with this char­ac­ter and the de­fault style would be the string "\x61\x00".

In re­al­i­ty, it’s a tad bit more com­pli­cat­ed than that, since one char­ac­ter may need more than one byte. I’m work­ing with UTF8, and my texts usu­al­ly con­tain non-ASCII char­ac­ters. In such a case, a char­ac­ter might span more than one cell, but the style byte is the same in all these cells. For ex­am­ple, the let­ter “ü”, rep­re­sent­ed as a UTF8 byte se­quence, would be "\xC3\xBC". If this let­ter should be styled using style num­ber 80, the style byte would be "\x50" (0x50 is the hex­adec­i­mal rep­re­sen­ta­tion of the num­ber 80). A byte se­quence con­tain­ing the cor­re­spond­ing two cells for the char­ac­ter “ü” with style 80 would be "\xC3\x50\xBC\x50".

So, a lexer takes a string with UTF8 bytes and re­turns a string with twice its length, with the orig­i­nal text and the style in­for­ma­tion mul­ti­plexed.

For most lan­guages, 32 dis­tinct styles are more than enough. To en­code one of 32 pos­si­ble styles, 5 bits are suf­fi­cient, and if we use a full byte for the style in­for­ma­tion, we have 3 ad­di­tion­al bits left for some aux­il­iary style in­for­ma­tion. For ex­am­ple, we could set one of those three ad­di­tion­al bits to in­di­cate that a word is mis­spelled, or that a line of source code is re­spon­si­ble for a run­time error, or what­ev­er we want to do with those ad­di­tion­al 3 bits.

As men­tioned above, HTML needs more styles, that’s why Scin­til­la uses 7 bits for the style in­for­ma­tion for 128 dis­tinct styles a token can have, leav­ing just one bit for other pur­pos­es.

Now I want two things: I want a func­tion that ac­cepts a nor­mal string of a text en­cod­ed as UTF8, and re­turns a string of cells with twice as much byte, con­tain­ing the styling in­for­ma­tion. And I want a func­tion that is able to split a text into HTML-chunks and TeX-chunks. Then I can split a text in chunks, feed those chunks in my styling func­tions, and then feed the styled strings back into my Styled­TextC­trl. Since, as men­tioned, wx­Python doesn’t ex­pose a method to lex in­di­vid­u­al strings, I cre­ate two in­vis­i­ble dummy Styled­TextC­trl el­e­ments.

First I cre­ate a class that in­her­its from Styled­TextC­trl:

I can’t use one of the pre­de­fined lex­ers, in­stead I have to pro­vide my own. I do this by set­ting the lexer of my el­e­ment to STC_LEX_CONTAINER. There­fore, since I’m using my own lexer, I can’t let Scin­til­la do the styling, but have to catch the EVT_STC_STYLENEEDED event my­self. And since I’m using more than 32 styles, I’ll have to tell Scin­til­la that I need 7 bits for my styling in­for­ma­tion.

Now I have to im­ple­ment my own lexer to style this el­e­ment:

I get the text of the el­e­ment, and now I have to split this text in chunks con­tain­ing pure HTML re­spec­tive­ly pure TeX.

At this point, I can’t avoid im­ple­ment­ing some­thing like a prim­i­tive pars­er. My pars­er needs to know when to switch be­tween “I’m cur­rent­ly read­ing HTML” states and “I’m cur­rent­ly read­ing TeX” states. It’s even more com­pli­cat­ed than that: I want to dif­fer­en­ti­ate be­tween TeX in round brack­ets and TeX in square brack­ets, and color them dif­fer­ent­ly.

The switch be­tween dif­fer­ent states is trig­gered by the pres­ence of de­lim­iters. But when I’m in HTML mode and in­side a tag, those de­lim­iters should be ig­nored. There­fore, I need at least two dif­fer­ent states for HTML: the de­fault state, and being in­side a tag.

The com­plete model of tran­si­tions from one state to the next when en­count­ing a de­lim­iter looks like this:

This means that if we are walk­ing along a string, and we are in de­fault HTML mode, and we en­counter the sub­string "\(", we switch to the round brack­et en­closed TeX state. The ad­di­tion­al pa­ram­e­ter True means that this is also a change of state that trig­gers a change of lex­ers.

Some code to ac­tu­al­ly per­form this pars­ing:

The de­lim­iters them­selves get their own chunks with own state flags, the sum of the state flags for the state be­fore and after the de­lim­iter, and a spe­cial flag for de­lim­iters. So, for ex­am­ple, "\(" gets its own chunk with flag STATE_DEFAULT + STATE_ROUND + STATE_DELIM. Since STATE_DEFAULT is 0, this is iden