If you’d just like to generate some lyrics, the models are hosted online at deepmetal.io. This approach worked quite well, and you can compare and contrast it with the earlier models here. The first character fed into the model at the beginning of generation is always a special start-of-sequence character.” – Iain We then sample from this distribution to get the next character in the sequence and this character goes on to become the next input to the model. At each step, we feed the current symbol into the model, and the model returns a probability distribution over the next character. “To generate text from the RNN model, we step character-by-character through a sequence. He unrolled the sequence for 32 characters and trained the model by predicting the next 32 characters, given their immediately preceding characters, while minimizing the mean cross-entropy: The RNN Iain used contains two LSTM layers of 512 units each, followed by a fully connected softmax layer. So he turned to recurrent neural network (RNN). Although it quickly beat the maximum likelihood Markov model, its longer outputs did not look like genuine heavy metal songs. His best try consisted of two feed forward layers of 512 units, followed by a softmax output, with layer normalisation, dropout and tanh activations, which he trained for 20 epochs to minimise the mean cross-entropy. He started out with feed forward networks on a character level. I lack the temporal, financial and computational resources to fully explore the hyperparameter space of these models, so the results presented here should be considered suboptimal.” – Iain On top of this, training the models here took days of computer time, plus more of my human time tweeking hyper parameters to get the models to converge. Even using powerful libraries, it took me a while to understand NNs well enough to use. “The maximum likelihood models we saw before took twenty minutes to code from scratch. He recognizes the power of neural nets, but says they also come at a cost: Iain used Keras, which ran on an amazon GPU instance. This model is based on observed word chains, for instance, those in the first two lines to Iron Maiden’s Number of the Beast:Īnother approach would be to train a neural network. Basically he develops a Markov Chain, in his opinion more of a “unsmoothed maximum-likelihood language model“, which determines the next most probable word based on the previous word(s). His first approach was to use probabilistic distributions known as language models. In part 2, Iain tried to create a heavy metal lyric generator (which you can now try out). He used the classification errors to examine which bands get confused with each other, and visualized this using two network graphs. He could predict the band behind a song by training a one-vs-rest logistic regression classifier based on the reduced lyric space of 150 dimensions after latent semantic analysis. Despite classifying a song to one of 120 different bands, the classifier had a precision and recall both around 0.3, with negligible hyper parameter tuning. While he also applied t-SNE to visualize band similarity in a two-dimensional space, the solution was uninformative due to low variance in the data. However, all his analysis worked out nicely. This allowed interesting analysis, such as a clustering of the various bands: Using cosine distance measures, Iain could compare the word vectors of the different bands, ultimately recognizing band similarity, and song representativeness for a band. This allowed him to contrast the word usage of the different bands, finding, for instance, one heavy metal band that was characterized by the words “ oh yeah baby got love“: fans might recognize either Motorhead, Machinehead, or Diamondhead. Lyric complexity relates positive to swearwords used.įurthermore, he ran some word importance analysis, looking at word frequencies, log-likelihood ratios, and TF-IDF scores. He applied a broad range of different analyses in Python, the code of which you can find here on Github.įor example, he starts part 1 by calculated the difficulty/complexity of the lyrics of each band using the Simple Measure of Gobbledygook or SMOG and contrasted this to the number of swearwords used, finding a nice correlation. Iain of wrote a three-piece series where he applied text mining to the lyrics of 222,623 songs from 7,364 heavy metal bands spread over 22,314 albums that he scraped from. This blog summarized work that has been posted here, here, and here.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |