How to do one hot encoding of unusual letters in matlab?

I have a table of german characters. I would like to one hot encode them so that i can input it to a neural network. How should i go about?

The problem is some of the characters are not accepted by matlab as characters. For example, 'ä' 'ö' 'ü' 'ß'

Regardless, I would like to know how to one hot encode any character from a TABLE in matlab.

Thanks in advance!

[Guillaume]

What does hot encoding mean?

[Mario Chiappelli]

Was ist heiss Codierung?

[Sanjana Sankar]

Please check the link provided by Walter about one-hot encoding

Answer 1

[Walter Roberson]

https://machinelearningmastery.com/why-one-hot-encode-data-in-machine-learning/ describes One-Hot Encoding (a term I was not aware of)

In MATLAB, see https://www.mathworks.com/help/deeplearning/ref/vec2ind.html

You might want to first construct a list of permitted characters, and map the input into an offset in that list. That will potentially save you from wasting bits on characters such as Œœ that you are not using.ŒŒ

[Sanjana Sankar]

Thank you! But my issue is focused on converting the letters to indices. I am not able to convert special characters like('ä' 'ö' 'ü' 'ß') to indices.

[Guillaume]

my issue is focused on converting the letters to indices

An index indicates the position of something in a set. What is the exhaustive list of the elements of the set you want to use? As Walter said, "you might want to first construct a list of permitted characters".

Once you know the set, it's trivial to convert your text into indices (ismember will do it).

Now having read about one-hot (I think wikipedia is clearer than Walter's link), converting to indices is not the same at all as one-hot encoding. On the other hand, unless you're programming FPGA or similar, I see no reason to use one-hot encoding. On a X86 processor, it's just a huge waste of memory

[Sanjana Sankar]

PFA file containing the characters that I want to one hot encode. Do you know how I can go about now? I am still having problems

[Walter Roberson]

permitted = ['A' : 'Z', 'a' : 'z', 'ä', 'ö', 'ü', 'ß', ' ', '.' ] ;

 [found, idx] = ismember(YourText, permitted);
  assert(all(found), 'unpermitted character detected')
  OneHot = ind2vec(idx) ;

[Sanjana Sankar]

OneHot = full(ind2vec(idx,max(idx)));

This is exactly what I was looking for and I got it from your code. Thank You! This works :)

Answer 2

[Guillaume]

file containing the characters that I want to one hot encode. Do you know how I can go about now

%phoneme_set: A cell array of phonemes to one-hot encode
assert(numel(phoneme_set) > 64, 'Cannot one-hot encode more than 64 phonemes with a 64-bit integer')
phoneme_set(:, 2) = num2cell(2 .^ uint64(0:size(phoneme_set, 1)-1))

[Sanjana Sankar]

But doesn't this give the value of 2^something? I want to save it as a vector with 0's and 1's. So that i can input this to an LSTM model.

[Guillaume]

number                     binary pattern (64 bits)
2^0    0000000000000000000000000000000000000000000000000000000000000001
2^1    0000000000000000000000000000000000000000000000000000000000000010
2^2    0000000000000000000000000000000000000000000000000000000000000100
...
2^63   1000000000000000000000000000000000000000000000000000000000000000

This is what one-hot encoding is. As I commented, this is useful for FPGAs and similar which operate at the bit level. On the generic processor of a computer, it's a complete waste of space but my answer does what you asked.

You could encode the one-hot encoded numbers that I generate as a vector of 0 and 1 (double) for even more waste of space:

phoneme_set(:, 3) = num2cell(fliplr(eye(size(phoneme_set, 1), 64)), 2)

Note that the binary pattern of the 0s in that encoding is 0000000000000000000000000000000000000000000000000000000000000000b and of the 1s is 0011111111110000000000000000000000000000000000000000000000000000b.

However, I suspect that what we have here is a XY problem. You have some unspecified problem doing something and you think you can solve it by using one-hot encoding (without really understanding what it means) so ask about one-hot encoding instead of your actual problem.

[Sanjana Sankar]

I have to feed characters to a neural network. And the most common method is one hot encoding. But I haven't found any examples on how it is done in MATLAB.