I have a couple of small projects I am working on around llms (#llm). I want to convert the llama2 project to java8 not java21 due to the prevalence of java8. Also want to build a llm around certain data.
I still haven't found a comprehensive guide on what is fully involved. This post is more about, "I confused like a 5 year old and want to highlight certain things"
These posts are the best and there is probably weeks worth of review of these videos and projects. And if you are thorough, you really should take apart, the tensor project. Tensorflow, pytorch, all of it to get a full understanding of the process. It is really just more than "run the inference" chat engine. And that is where I got stuck on. They say, all you need is the C file and the token data and model data. That is true but there are a lot of questions I asked.
Where are the actual tokens? Where did you get them from? The actual words?
What about the model data? the numbers?
One thing that wasn't clear from the code or various posts. We obviously got the base vocabulary from somewhere but where. This review is based on the TinyStories set which can be found in this article.
See this article on
https://arxiv.org/abs/2305.07759
https://arxiv.org/pdf/2305.07759
Also see paper
https://arxiv.org/abs/2203.15556
"Training Compute-Optimal Large Language Models" -- Deepmind, 2022
Download
The file is about 1.5 gigs
And hugging face downloads
https://huggingface.co/datasets/roneneldan/TinyStories
And I still wasn't clear how they fed the vocab into processing. The JSON looks like this.
Finally, here is the actual text.
[
{
"story":"\n\nLily and Ben are friends. They like to play in the park. One day, they see a big tree with a swing. Lily wants to try the swing. She runs to the tree and climbs on the swing.\n\"Push me, Ben!\" she says. Ben pushes her gently. Lily feels happy. She swings higher and higher. She laughs and shouts.\nBen watches Lily. He thinks she is cute. He wants to swing too. He waits for Lily to stop. But Lily does not stop. She swings faster and faster. She is having too much fun.\n\"Can I swing too, Lily?\" Ben asks. Lily does not hear him. She is too busy swinging. Ben feels sad. He walks away.\nLily swings so high that she loses her grip. She falls off the swing. She lands on the ground. She hurts her foot. She cries.\n\"Ow, ow, ow!\" she says. She looks for Ben. She wants him to help her. But Ben is not there. He is gone.\nLily feels sorry. She wishes she had shared the swing with Ben. She wishes he was there to hug her. She limps to the tree. She sees something hanging from a branch. It is Ben's hat. He left it for her.\nLily smiles. She thinks Ben is nice. She puts on his hat. She hopes he will come back. She wants to say sorry. She wants to be friends again.",
"instruction":{
"prompt:":"Write a short story (3-5 paragraphs) which only uses very simple words that a 3 year old child would understand. The story should use the verb \"hang\", the noun \"foot\" and the adjective \"cute\". The story has the following features: the story should contain at least one dialogue. Remember to only use simple words!\n\nPossible story:",
"words":[
"hang",
"foot",
"cute"
],
"features":[
"Dialogue"
]
},
"summary":"Lily and Ben play in the park and Lily gets too caught up in swinging, causing Ben to leave. Lily falls off the swing and hurts herself, but Ben leaves his hat for her as a kind gesture.",
"source":"GPT-4"
},
See my code here
https://github.com/berlinbrown/llama2-java8.java
Comments
Post a Comment