I got the invalid character portion of this challenge without any issue, but I’m stuck on the token portion…
catch Parser.Error.invalidToken(let token) {
let distanceToInvalidToken = input.distance(from: input.startIndex, to: input.firstIndex(of: [character associated to this token])
// how do I get the original value from input that the token represents?
print("Invalid token during parsing at index \(distanceToInvalidToken): \(token)")
I don’t have the book, but more information should be available at the point the error is detected. You can take advantage of that and raise an error exception that includes the required information.
The way Token is defined, you can’t be guaranteed to have access to that since it’s conditional. If the token is a .number you can, but if it’s a .plus you cannot. You’d have to do something like this:
var value: String
switch token {
case let .number(digit):
value = String(digit)
case .plus:
value = "+"
case .minus:
value = "-"
}
But the way you’re using that won’t work. Consider the example from the book: “10 + 3 3 + 7”. The invalid token is “3” but it isn’t the first “3” in the input string, so using firstIndex(of:) won’t give you the position of the invalid token. When invalidToken is thrown you’ll need to include the position of the token alongside the token itself, so you’ll need to save that information when the token is created in Lexer.
I’m don’t understand where or how to implement your switch statement.
As for trying to save the information in Lexer, I’ve also tried adding a new variable:
class Lexer {
...
var errorPosition: String.Index
...
init(input: String) {
self.input = input
self.position = input.startIndex
self.errorPosition = input.startIndex
}
...
func lex() throws -> [Token] {
var tokens = [Token] ()
while let nextCharacter = peek() {
switch nextCharacter {
...
default:
// Something unexpected - need to send back an error
errorPosition = input.index(after: position) // silver challenge - trying to capture the position of the error character...doesn't position change as you advance() within the other cases?
throw Lexer.Error.invalidCharacter(nextCharacter)
}
}
But with my output, the errorPosition never changes:
Evaluating: 10 + 3 3 + 7
12
Lexer output: [Number: 10, Symbol: +, Number: 3, Number: 3, Symbol: +, Number: 7]
Invalid token during parsing at index 0: Number: 3
You would put it in the code that’s catching the error:
} catch Parser.Error.invalidToken(let token) {
var badChar: String
switch token {
case let .number(digit):
badChar = String(digit)
case .plus:
badChar = "+"
case .minus:
badChar = "-"
}
let distanceToInvalidToken = input.distance(from: input.startIndex, to: input.firstIndex(of: badChar))
print("Invalid token during parsing: \(token) at position: \(distanceToInvalidToken)")
But as I said, that doesn’t always find the right character.
As for your new code, it isn’t working because you put it in the wrong place. The error you’re getting is invalidToken which is thrown from parse(); lex() isn’t throwing invalidCharacter because there’s no invalid character in the input string so the code you added to lex() never gets called. lex() needs to save the location of all the tokens, so when parse() throws invalidToken it can look up where the bad token was located.
Thanks for clarifying.
I see what you’re saying about not finding the right character all the time.
This isn’t correct:
Evaluating: 10 + 3 3 + 7
Lexer output: [Number: 10, Symbol: +, Number: 3, Number: 3, Symbol: +, Number: 7]
Invalid token during parsing Number: 3 at index 5.
Though, this is:
Evaluating: 10 + 3 8 + 7
Lexer output: [Number: 10, Symbol: +, Number: 3, Number: 8, Symbol: +, Number: 7]
Invalid token during parsing Number: 8 at index 7.
I also got a compile error with my distanceToInvalidToken stating
rror: cannot convert value of type 'String' to expected argument type 'String.Element' (aka 'Character')
But once I convert it to Character, it crashes when I replace ‘3’ with ‘33’.
I still don’t understand why it’s so difficult to simply capture the exact character’s exact index to calculate an accurate distanceToInvalidToken.
Right, firstIndex(of:) only works on individual characters, you can’t pass in a string. If you want to find the location a token that’s longer than a single character, you’d have to do something like:
let r = input.range(of: badChar) // where badChar is a String
let distanceToInvalidToken = input.distance(from: input.startIndex, to: r!.lowerBound)
which would give you the location of the first character in the first occurrance of badChar.
As I suggested before, it’s easier to do this calculation inside lex() because it knows which characters it’s reading for each token. You don’t have to search for anything
I’m trying to do this in lex(), though it’s very inefficient…
I’ve brought back my errorPosition variable and am calculating this within each case of my switch nextCharacter
errorPosition = input.distance(from: input.startIndex, to: position)
Now, I can capture the position of the number associated to the token. But now how do I make it available to the token?
Also, was this easier for everyone else on these forums to complete? I can’t be the only one struggling with these challenges.
If it’s easier for you, I’d be interested in learning more from anyone not named @JonAult about how you tackle these more easily than I’m trying to.
Update: I’ve re-read the chapter but am nowhere near the correct solution to this challenge.
This is beyond frustrating.
If you were as frustrated as I am, how did you overcome the frustration to reach the solution?
If this was an easy challenge for you, then why was it easy, and what was your step-by-step approach to reaching the solution?
Just got a solution. It took me some hours on a few days (3 days ?). I don’t know if it is the intended solution but it works. I knew JonAult was right stating:
The real problem was to find a way to do that. I said to myself that instead of returning only an array of Token (that is [Token]), one must return a connected pair of information: that is a token with an associated String.Index. I decided to try having func lex() to return an array of tupples [(Token, String.Index)] like:
func lex() throws -> [(token: Token, stringIndex: String.Index)] {
var tokensWithPositions = [(token: Token, stringIndex: String.Index)]()
while let nextCharacter = peek() {
switch nextCharacter {
case "0" ... "9":
let value = getNumber()
tokensWithPositions.append((.number(value), input.index(before: position)))
...
so Parser.Error.invalidToken could throw a tokenWithPosition tupple.
After these changes, I had to work on eliminating errors found by Xcode. There were a lot of them. In Parser for instance, cases in switch statements had to use tupples like :
case (.plus, tokenWithPosition.stringIndex):
let nextNumber = try getNumber()
value += nextNumber
and I had to add a default case throwing an invalidToken error.
Then, it worked. I still had to make some adjustments (count of character was different wether the faulty character was a number or a sign (+ or -) and printing the Lexer output had plenty of Swift.String.Index [an easy fix using map to print only the tokens as was previously done]) but everything looks OK now.
That was hard to do, I must admit, but no new concept was involved. It is just that a seemingly simple modification had plenty of implications. But, as what had been said elsewhere in the book, one just has to follow indications from the debugger.
Keep going if you want, you could be rewarded. Success is fun.
Thanks.
I’ve been stuck on it for three weeks, not three days
I don’t know why I couldn’t think of using a tuple.
I’m trying to graft your solution, but I’m still seeing errors.
Like here in my parser:
let parser = Parser(tokens: tokens) // Error: Cannot convert value of type '[(token: Token, stringIndex: String.Index)]' to expected argument type '[Token]'
What are the exact steps I need to THINK about to resolve the error and arrive at the right solution?
If this is supposed to take days to complete successfully, I don’t want to spend another three weeks trying to do so.
Just relax and read that error message carefully 
The message is saying that the Parser is being given an array of tuples of the form (Token, String.Index) while it is expecting only an array of Tokens. So, either give the Parser an array of Tokens or modify it so that it can accept an array of tuples of the form (Token, String.Index).
just relax and read the error message carefully