This is not the document you are looking for? Use the search form below to find more!
About Tokens and Lexemes
- Added: May, 12th 2011
- Reads: 133
- Downloads: 0
- File size: 487.98kb
- Pages: 28
- content preview
- Name: henriette
What People Need to Know about Acne and Its Solution E-Book
by: iHealthiLicious, 46 pagesWhat People Need to Know about Acne and Its Solution E-Book
What I need to know about Eating and Diabetes
by: imogen, 52 pagesWhat I need to know about Eating and Diabetes. A guide to people with diabetes.
Learning More About Bankruptcy And Debt Problems
by: James, 1 pagesLearning More About Bankruptcy And Debt Problems
Top 7 myths about Israel and zionism
by: raceandreligion, 2 pagesTop 7 myths about Israel and zionism
Debunking Some Myths About Acne and Diet.
by: jswilkinson, 2 pagesThere seems to be a lot of cautionary tales about what you should and what you should not eat when dealing with acne. Are these just 'old wives tales' or is there some truth in the matter
All You Need to Know About Buying and Selling Gold and Were Afraid to Ask!
by: joshuacarson1231, 2 pagesSelling Gold For Profit can be done in three different ways: The first, as an individual having no interest in investing, and only interested in making small amounts of cash. The second, as a serious ...
Truth About Cataracts And Its Effects
by: MagmireToby, 2 pagesCataract eye drops - Bright Eyes NAC eye drops cataracts trial on TV featuring Ethos Endymion Bright Eyes NAC (N-Acetyl-Carnosine) eye drops for cataracts, glaucoma and macular degeneration (AMD).
Facts About Bees and Africanized Bees
by: rika, 3 pagesHoney bees are not native to the North America. Honey bees currently pollinate about 90 agricultural crops (accounting for 80% of the pollination in the US): ~ $10 billion pollination business, ...
Getting Our History Right: Six Errors about Darwin and His Influence
by: shinta, 18 pagesThe Darwin Exhibition created by the American Museum of Natural History is the centerpiece of the bicentennial of Darwin’s birth. It opened in November 2005 and will circulate to a ...
Externalism about Content and McKinsey-style Reasoning
by: monkey, 38 pagesIt’s widely accepted nowadays that the contents of some of our thoughts are externalist: we’re only able to have thoughts with those contents because we inhabit environments ...
- About tokens and lexemes Ben Scholzen Game Developer Gameforge Productions GmbH
- What we'll cover
- Definition of a compiler, tokenizer and parser
- Basic structure of a tokenizer and a parser
- Where to optimize things for PHP
- What about parser generators?
- They are evil!
- PHP_LexerGenerator, PHP_ParserGenerator, lemon-PHP
- Create lots of function calls like lemon parsers in C
- Are not working very performance-wise
- Will eat up all your memory
- Conclusion
- Don't use them!
- Let's get started
- What a compiler is and how it works
- Acts as frontend for the application
- Converts human-readable data into machine-readable data
- Consists of a two components:
- The lexer:
- Is a finite-state-machine
- Reads the input stream
- Clears up the input data
- Creates a list of tokens
- The parser:
- Gets tokens from the tokenizer
- Converts them into a data structure
- What a compiler is and how it works Lexer Parser Tokens Document Stream Structure
- Sounds great, but where do I need it?
- Formatting languages
- BB-Code
- Wiki-Codes
- Description languages
- iCalendar / vCalendar
- XML
- Even programming languages
- JavaScript
- PHP
- Anything else you want your program to understand
- The lexer (or tokenizer)
- What are tokens?
- Categorized block of text
- Token type
- Corresponding block of text (lexeme)
- List of tokens represents an entire document
- Example in PHP: $value = 5 * 7 ;
- How the tokenizer works
- Define possible states of the lexer
- Tokenize the input in a loop
- Scan with preg_match()
- Strtok() is mostly too simple
- Reading char-by-char is too slow
- Use the offset parameter
- Use the G assertion (^ won't work)
- Always store the current position
- Use either a switch-statement or a structured array
- Return the tokens
- What we can optimize
- Use little memory
- Always just read a partial part of the document into memory
- Via fopen() and fgets()
- Requires previous knowledge about when tokens end
- Offer a method for the parser to get a partial bunch of tokens
- Speed up execution-time
- Do no internal function-calls if applicable
- Going into practice
- The beginning
- Use little memory
- Via fopen() and fread()
- Requires previous knowledge about when tokens end
- Offer a method for the parser to get a partial bunch of tokens
- Speed up execution-time
- Do no internal function-calls if applicable
- Throwing in a file
- Preparing stuff
- Base state
- Operator state
- Value state
- Rounding it up
- Some actual testing
- And what we get
- array(6) {
- [0]=>
- array(2) {
- [0]=>
- string(8) "variable"
- [1]=>
- string(6) "$value"
- }
- [1]=>
- array(2) {
- [0]=>
- string(8) "operator"
- [1]=>
- string(1) "="
- }
- [2]=>
- array(2) {
- [0]=>
- string(6) "number"
- [1]=>
- string(1) "5"
- }
- [3]=>
- array(2) {
- [0]=>
- string(8) "operator"
- [1]=>
- string(1) "*"
- }
- [4]=>
- array(2) {
- [0]=>
- string(6) "number"
- [1]=>
- string(1) "7"
- }
- [5]=>
- array(2) {
- [0]=>
- string(8) "operator"
- [1]=>
- string(1) ";"
- }
- }
- The parser
- So we have a bunch of tokens, what now?
- Loop through the tokens and analyze them
- Create an object-oriented tree-structure or interpret
- Avoid non-tail recursion
- Use tail-recursion (trampoline) instead
- Saves you from hitting the stack limit
- That's it!
- Summary — Questions?
- Where to go from here
- Wikipedia: http://en.wikipedia.org/wiki/Compiler http://en.wikipedia.org/wiki/Parsing
- About tail-recursion in PHP: http://www.alternateinterior.com/2006/09/tail-recursion-in-php.html
- My blog: http://www.dasprids.de
- Rate this talk: http://joind.in/635
- Follow me on twitter:
- http://www.twitter.com/dasprid
- Thank you!
Add New Comment