QCodeEdit

**Methedrine** · 8th December 2007, 21:37

Originally Posted by fullmetalcoder

Fixed both.

Thanks again :-)

Originally Posted by fullmetalcoder

Fixed both.

Why don't you want to use the default syntax engine? Wouldn't it be simpler and better to extend it rather than creating another one from scratch ?
What exactly do you expect it do do and how much would its internal structure differ from the one of the default syntax engine?

Answers to these would probably help me to change the syntax engine interface to make room for more than one without changing the existing one...

We already wrote one from scratch before you came around with this great edit control. It works like a charm, so far (thread-safe, DFA based, result caching). Qtcentre-member kaos can tell you more about it since, he wrote our parser.
I am also certain that he can give you a better answer to your second question.

**allstar** · 9th December 2007, 10:33

New bug: Hold Ctrl and select a few lines of text with mouse.

**fullmetalcoder** · 9th December 2007, 10:41

Originally Posted by Methredine

We already wrote one from scratch before you came around with this great edit control. It works like a charm, so far (thread-safe, DFA based, result caching). Qtcentre-member kaos can tell you more about it since, he wrote our parser.
I am also certain that he can give you a better answer to your second question.

I was thinking about turning my current engine into a DFA based one so if you have one working already it could save me that pain.

What format do you use for language definitions? What data do you store for each line of highlighted text and how? And may I have a look at your code?

Originally Posted by allstar

New bug: Hold Ctrl and select a few lines of text with mouse.

It's not a bug, it's a feature...

If you read the previous posts you'll notice we talked about "cursor mirrors" "column edit" and "column selection"...

**kaos** · 9th December 2007, 16:33

Originally Posted by fullmetalcoder

I was thinking about turning my current engine into a DFA based one so if you have one working already it could save me that pain.

What format do you use for language definitions? What data do you store for each line of highlighted text and how? And may I have a look at your code?

We currently use some plain text format. Creating a parser which reads xml instead of text would be quite easy. But i personally don't like xml definitions for these kind of things (escaping of <>&, simple key: values are easier to read for these kind of things).

The lexer itself only emits a list of tokens for the parsed text. The executor currently works quite much line based (and stores tokens line as well). When text gets changed in the edit control the lexer thread is notified of that change and reparses the changed line. When it reaches the end of the line it looks into a cache to see if the state at the end of line matches the state at the end of the line before the change occured. If that is the case it only reparses the changed line (works quite well, i haven't seen it parse anything more then needed yet). I saw some caching in your code as well, but that seemed to be context and not line based. In general i think one could map your xml definition to that lexer with some changes. One problem could be mapping your context to my groups.

I really like the approach of parsing the entire thing in a separate thread. This way typing always stays responsive.
So one requirement would be that the text retrieval methods should be reentrant. And replacing the highlighter would beed to be possible of course. The only thing required is a callback when text has changed i think.

Drop me a pm and i can hook you up with the source. It will be released eventually anyways, it's just not in a state where we think that we should release it.

**fullmetalcoder** · 9th December 2007, 17:17

Originally Posted by kaos

We currently use some plain text format. Creating a parser which reads xml instead of text would be quite easy. But i personally don't like xml definitions for these kind of things (escaping of <>&, simple key: values are easier to read for these kind of things).

XML does induce some nasty escaping and a more complex parsing (even though this may not remain always true if you use QXmlStreamReader) but on the other hand it allows an easy and extremely powerful hierarchization and plain text can not compete with XML in that area...

Originally Posted by kaos

The lexer itself only emits a list of tokens for the parsed text.

How do you map a token with a given text format? Is that handled within the lexer which sneds the info along the token?

Originally Posted by kaos

I saw some caching in your code as well, but that seemed to be context and not line based. In general i think one could map your xml definition to that lexer with some changes. One problem could be mapping your context to my groups.

the "caching" is line based but instead of using an additional state variable I just check the context stack. Could you define what a "group" is in your syntax engine?

Originally Posted by kaos

I really like the approach of parsing the entire thing in a separate thread. This way typing always stays responsive.

Even with big files (about 1meg of C++ code) QCodeEdit remains responsive, unless you're nasty enough to place a multiline comment at the begining of such a big file and switch between commented and uncommented... But even this manipulation doesn't alter the responsiveness on "normal" files ( ~3-4k lines)

Some people thing that threads make things faster but they actually make them way slower... The only advantage is responsiveness but what if responsiveness can be preserved without threading? The first version of Edyuk completion engine was threaded and it was SLOW (a couple of secs to display results). As soon as I dropped threading and replaced QListWidget by a QListView with a custom model it fell to a couple of msecs!!! It made me think a little more about threading and now I only use it when the speed loss is irrelevant in front of the responsiveness... In the case of highlighting I don't think threading is such a good idea but maybe some benchmarks could prove me wrong...

**elcuco** · 9th December 2007, 17:59

Can you demostrate in a small demo your claim? Yes, using another thread has it's overhead, but it should not be of another magnitude.

Another thing I wanted to ask, is what happens when you paint the document in a separate thread, and meanwhile a user modifies the document. But I think it's not a valid question anymore.

**fullmetalcoder** · 9th December 2007, 18:15

Originally Posted by elcuco

Can you demostrate in a small demo your claim? Yes, using another thread has it's overhead, but it should not be of another magnitude.

If you fetch Edyuk 0.8.x or 0.7.x( I don't remember when actually I made the change...) you'll see a pretty slow completion and you can compare with what happens now

Originally Posted by elcuco

Another thing I wanted to ask, is what happens when you paint the document in a separate thread, and meanwhile a user modifies the document. But I think it's not a valid question anymore.

I don't know to whom you ask that question but I'd say it's never been a valid question at all... kaos and methredine clearly pointed out that their engine is thread safe and rely on signals/slots to request painting updates and QCodeEdit never relied on threading... Besides your question is a bit unclear as "painting" is impossible from a thread... The most you can do is updating formatting informations (in a thread safe way or it'll segfault).

**kaos** · 9th December 2007, 18:33

Originally Posted by fullmetalcoder

XML does induce some nasty escaping and a more complex parsing (even though this may not remain always true if you use QXmlStreamReader) but on the other hand it allows an easy and extremely powerful hierarchization and plain text can not compete with XML in that area...

Yes, i am a big fan of xml. But i found a custom format in that case much more managable.

How do you map a token with a given text format? Is that handled within the lexer which sneds the info along the token?

Each token gets a unique id when the lexer file gets parsed. With this unique id several infos about the token are saved (the name, the language, etc)

the "caching" is line based but instead of using an additional state variable I just check the context stack. Could you define what a "group" is in your syntax engine?

as far as i can see it's quite much the same as your context. it defines a certain "block" with a start and an end and additional rules for the text in the group.

Even with big files (about 1meg of C++ code) QCodeEdit remains responsive, unless you're nasty enough to place a multiline comment at the begining of such a big file and switch between commented and uncommented... But even this manipulation doesn't alter the responsiveness on "normal" files ( ~3-4k lines)

That's the part where the multithreaded model excels quite much. One other reason for me to go the threaded part were the emerging multithreaded/multicode cpus

Some people thing that threads make things faster but they actually make them way slower... The only advantage is responsiveness but what if responsiveness can be preserved without threading? The first version of Edyuk completion engine was threaded and it was SLOW (a couple of secs to display results). As soon as I dropped threading and replaced QListWidget by a QListView with a custom model it fell to a couple of msecs!!! It made me think a little more about threading and now I only use it when the speed loss is irrelevant in front of the responsiveness... In the case of highlighting I don't think threading is such a good idea but maybe some benchmarks could prove me wrong...

Well, making things threaded is of course more complicated then easy to get right. The lexer definitly doesn't need to be multi threaded, but i since the solution works really well, i saw no reason in changing that. I would not do the completer multithreaded tho, since the user usually expecting a response when he triggers auto completion. But for me personally any delay when typing is just not acceptable in an editor. And I didn't see any problem with lexing the stuff in an extra thread. I've ran my code on an p3 450 and it still worked rather smooth. I haven't benchmarked that entire thing tho, but iirc most of the time the lexer thread already was done with the line before the control went redrawing itself (on a single core/thread cpu)

**kaos** · 9th December 2007, 18:36

Originally Posted by elcuco

Can you demostrate in a small demo your claim? Yes, using another thread has it's overhead, but it should not be of another magnitude.

Another thing I wanted to ask, is what happens when you paint the document in a separate thread, and meanwhile a user modifies the document. But I think it's not a valid question anymore.

The drawing and the user input usually happen in the same thread. But it still is a valid question since one worker thread could possibly modify the text while drawing. So if you make such a control to the thread safe all the text modification methods need to be properly protected against with a mutex, etc

**fullmetalcoder** · 9th December 2007, 18:41

Originally Posted by kaos

Each token gets a unique id when the lexer file gets parsed. With this unique id several infos about the token are saved (the name, the language, etc)

But then how do you handle parsing numbers for instance? It must be done with a more complex DFA than a keyword and allow for many more than one single token... I guess you meant that the id is associated with the DFA matching state or am I missing something?

**kaos** · 9th December 2007, 18:53

Originally Posted by fullmetalcoder

But then how do you handle parsing numbers for instance? It must be done with a more complex DFA than a keyword and allow for many more than one single token... I guess you meant that the id is associated with the DFA matching state or am I missing something?

Ah, yeah.
Well, all the tokens get compiled into one big automaton per group where the final state carries information which token id to emit when the match completes at that final state.

**Usability** · 12th December 2007, 21:53

I just noticed the license syntax selection is fooled by dots in the file name.
For example

example/example lib/qpanellayout.cpp

produces a correctly colourcoded file, while

cp lib/qpanellayout.cpp a.1.cpp
example/example a.1.cpp

Shows no colourcoding at all. I think the extension is the text after the final dot in a file name.

**fullmetalcoder** · 14th December 2007, 14:03

Originally Posted by Usability

I just noticed the license syntax selection is fooled by dots in the file name.
For example

example/example lib/qpanellayout.cpp

produces a correctly colourcoded file, while

cp lib/qpanellayout.cpp a.1.cpp
example/example a.1.cpp

Shows no colourcoding at all. I think the extension is the text after the final dot in a file name.

The issue must come from a use of QFileInfo::completeSuffix()... I'll correct it ASAP.

**Usability** · 18th December 2007, 12:02

How do you feel about making ctrl-f always take you to the find box?
Currently you have to first check if the find box is open, because if it is, you need to key two ctrl-f to get there. (The first one just closes the box, the second opens it again and takes you in).

I think it would be much easier this way. Conventionally you use escape to leave a dialog, not a repeat of the key that opened it.

**fullmetalcoder** · 19th December 2007, 17:03

Originally Posted by Usability

How do you feel about making ctrl-f always take you to the find box?

I'll have that sorted out ASAP.

BTW, the suffix issue has been fixed in case you didn't notice.

**travlr** · 26th December 2007, 16:30

Hi fmc,
A couple of issues/observations for edyuk.

First, I don't understand... Is not the trunk in SVN repo not the most recent code??? Huh? This quite unorthodox, isn't it. I've never seen this before. Quote from the website:

...and, as many people dislike (or don't even know) SVN, I ended up releasing a new package to avoid confusions with the very old one left on Sourceforg donwloads servers...

You can get it in the downloads section and enjoy all the improvements.

The ./build script reports a syntax error

Qt Code:

Switch view

oxbo% ./build -h
./build: 60: Syntax error: Bad substitution

oxbo% ./build -h
./build: 60: Syntax error: Bad substitution

To copy to clipboard, switch view to plain text mode

Using qt-4.4, a Make error regarding QAtomic and QBasicAtomic (4.4 has QAtomicInt and QAtomicPointer, Before 4.4 there is no "QAtomic*" at all according to the class refs)

Qt Code:

Switch view

In file included from ../../../../src/edyuk-1.0.0-pre1/3rdparty/qcodeedit2/lib/qeditor.cpp:19:
../../../../src/edyuk-1.0.0-pre1/3rdparty/qcodeedit2/lib/document/qdocument_p.h:32:19: error: QAtomic: No such file or directory
../../../../src/edyuk-1.0.0-pre1/3rdparty/qcodeedit2/lib/document/qdocument_p.h:215: error: â€˜QBasicAtomicâ€™ does not name a type
../../../../src/edyuk-1.0.0-pre1/3rdparty/qcodeedit2/lib/document/qdocument_p.h: In member function â€˜void QDocumentCursorHandle::ref()â€™:
../../../../src/edyuk-1.0.0-pre1/3rdparty/qcodeedit2/lib/document/qdocument_p.h:202: error: â€˜m_refâ€™ was not declared in this scope
../../../../src/edyuk-1.0.0-pre1/3rdparty/qcodeedit2/lib/document/qdocument_p.h: In member function â€˜void QDocumentCursorHandle::deref()â€™:
../../../../src/edyuk-1.0.0-pre1/3rdparty/qcodeedit2/lib/document/qdocument_p.h:203: error: â€˜m_refâ€™ was not declared in this scope
../../../../src/edyuk-1.0.0-pre1/3rdparty/qcodeedit2/lib/document/qdocument_p.h:203: error: â€˜m_refâ€™ was not declared in this scope
../../../../src/edyuk-1.0.0-pre1/3rdparty/qcodeedit2/lib/qeditor.cpp: In member function â€˜virtual bool QEditor::moveKeyEvent(QDocumentCursor&, QKeyEvent*, bool*)â€™:
../../../../src/edyuk-1.0.0-pre1/3rdparty/qcodeedit2/lib/qeditor.cpp:1973: warning: unused variable â€˜prevcolâ€™
make[2]: *** [.build/4.4.0-tp1-unix/obj/release/qeditor.o] Error 1
make[2]: Leaving directory `/d/bld/edyuk/src/lib'
make[1]: *** [release] Error 2
make[1]: Leaving directory `/d/bld/edyuk/src/lib'
make: *** [sub-src-lib-lib-pro-make_default] Error 2
makeobj[0]: Leaving directory `/d/bld/edyuk'

In file included from ../../../../src/edyuk-1.0.0-pre1/3rdparty/qcodeedit2/lib/qeditor.cpp:19:
../../../../src/edyuk-1.0.0-pre1/3rdparty/qcodeedit2/lib/document/qdocument_p.h:32:19: error: QAtomic: No such file or directory
../../../../src/edyuk-1.0.0-pre1/3rdparty/qcodeedit2/lib/document/qdocument_p.h:215: error: â€˜QBasicAtomicâ€™ does not name a type
../../../../src/edyuk-1.0.0-pre1/3rdparty/qcodeedit2/lib/document/qdocument_p.h: In member function â€˜void QDocumentCursorHandle::ref()â€™:
../../../../src/edyuk-1.0.0-pre1/3rdparty/qcodeedit2/lib/document/qdocument_p.h:202: error: â€˜m_refâ€™ was not declared in this scope
../../../../src/edyuk-1.0.0-pre1/3rdparty/qcodeedit2/lib/document/qdocument_p.h: In member function â€˜void QDocumentCursorHandle::deref()â€™:
../../../../src/edyuk-1.0.0-pre1/3rdparty/qcodeedit2/lib/document/qdocument_p.h:203: error: â€˜m_refâ€™ was not declared in this scope
../../../../src/edyuk-1.0.0-pre1/3rdparty/qcodeedit2/lib/document/qdocument_p.h:203: error: â€˜m_refâ€™ was not declared in this scope
../../../../src/edyuk-1.0.0-pre1/3rdparty/qcodeedit2/lib/qeditor.cpp: In member function â€˜virtual bool QEditor::moveKeyEvent(QDocumentCursor&, QKeyEvent*, bool*)â€™:
../../../../src/edyuk-1.0.0-pre1/3rdparty/qcodeedit2/lib/qeditor.cpp:1973: warning: unused variable â€˜prevcolâ€™
make[2]: *** [.build/4.4.0-tp1-unix/obj/release/qeditor.o] Error 1
make[2]: Leaving directory `/d/bld/edyuk/src/lib'
make[1]: *** [release] Error 2
make[1]: Leaving directory `/d/bld/edyuk/src/lib'
make: *** [sub-src-lib-lib-pro-make_default] Error 2
makeobj[0]: Leaving directory `/d/bld/edyuk'

To copy to clipboard, switch view to plain text mode

This was just a first look, before investigating reasons or fixes.

Thank you,
-travlr

**Usability** · 28th December 2007, 15:27

Double click in the editor got broken recently. With a slightly older QCodeEdit, double clicking selects a word. In the latest svn, double click does nothing.

**fullmetalcoder** · 29th December 2007, 17:14

@travlr :

There must be a misunderstanding somehow... SVN repo always hold the most recent code AND from time to time I package the content of the repo and put in on Sf.net dowload servers so that more people can play with it (as some are reluctant to use SVN...)
Would you be running under BSD? Or some exotic UNIX system? I must admit I'm not an expert of clean shell scripting and I'd be able if someone could help me to improve the portability of the build script...
Do you know what I need to do in order to make it compatible with Qt 4.4 then? Looks like it's just a matter of include files. Or did the QBasicAtomic class disappear completely in the utter nothingness?
Some of your comments concern Edyuk ad not QCodeEdit so it may be better to post in the appropriate thread.

@Usability : I fixed the double click regression.

**Usability** · 30th December 2007, 11:31

Double click working again, thanks!

**fullmetalcoder** · 3rd January 2008, 17:40

Originally Posted by Usability

How do you feel about making ctrl-f always take you to the find box?

Done.

A couple other changes have happened lately. Among which the addition of a line change panel which indicates the modified lines since last save.

Thread: QCodeEdit

Thread Tools

Search Thread

Display

Re: QCodeEdit

Re: QCodeEdit

Re: QCodeEdit

Re: QCodeEdit

Re: QCodeEdit

Re: QCodeEdit

Re: QCodeEdit

Re: QCodeEdit

Re: QCodeEdit

Re: QCodeEdit

Re: QCodeEdit

Re: QCodeEdit

Re: QCodeEdit

Re: QCodeEdit

Re: QCodeEdit

Re: QCodeEdit

Re: QCodeEdit

Re: QCodeEdit

Re: QCodeEdit

Re: QCodeEdit

Bookmarks

Bookmarks

Posting Permissions